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This publication should be read as a workbook. It is conceived as a catalog without linear 
order, as a conglomeration of diverse texts and images that relate and react to one another 
in multivarious ways. Sometimes they contradict one another, but taken together, they are 
intended to trigger discussion about the multitude of experiences with actors, avatars, digital 
assistants, and other artificial beings bestowed upon us by new technologies. The images and 
texts seek to spur new approaches to scientific, artistic, and aesthetic engagement with such 
artificial beings, which includes animate technological artefacts with humanlike faces and 
gestures, androids and robots that, while perceptibly dissimilar to humans, are supposed to 
interact with us. We meet them with acceptance or reject and fear them. The contributions 
bring them into communication with people, and theatrical actors in particular, in order to 
study the social relationships between humans, machines, and their “descendants.” The cata- 
log draws on a longstanding collaboration between three very different research groups: the 
Institute for Theory and the Institute for Performing Arts and Film at the Zurich University 
of the Arts, which work with the aesthetics, forms, dramaturgy, and production of artificial 
scenarios, as well as the Swiss Epilepsy Clinic, which conducts neuroscientific research. In 
the run of the collaboration, all three partners pursued their own methodological strate- 
gies, from philosophy to approaches from artistic research and visualization techniques to 
EEG tests and fMRI scans. But all three were united in the goal of seeking to understand 
how humans react to artificial beings, and in particular the differences between encounters 
with human faces and encounters with artificially generated faces, such as those of avatars, 
animated actors, and the like. The catalog brings together the findings of this collaborative 
research. It also contains contributions from other partners, important reference texts, and 
further supplementary materials. 

The research focused on how people perceive and cognitively process perceptions 
of real people and artificial artefacts and objects that represent people, whether it be in 
image form or by means of other visualizations, on the stage, in photorealistic paintings, and 
in film. The inquiry sought to uncover systematic, practical, and empirical evidence—or a 
lack thereof—about whether we perceive such representations differently than we do living 
people, especially when both only appear to us in images or recordings. In order to ensure 
that the different research projects were comparable with one another, they were limited 
to perceptions of faces. The key variables were actors who mimed certain facial expressions 
perceived as signifying emotions, and artificial beings like avatars that, by means of real-time 
motion capture, imitate the same expression. Facial recognition programs were used to 
scan the actors’ faces and transform them into avatars. The programs included that of the 
Zurich start-up Faceshift, which was acquired by Apple during the course of the research, 
and FaceRig from Romania’s Holotech Studios SRL." The initial hypothesis was that human 
reactions to human faces, on the one hand, and to avatars, on the other, represent distinct 
forms of relating that do not just concern perception, but extend deep into the world of human 
emotions. In short, the assumption was that people register these impressions of faciality 
in disparate, incommensurable ways. If the hypothesis is correct, it would problematize, if 
not outright undermine the substitution theory of simulation, which holds that, given the 
necessary technological development, simulated beings can be perfected to such a degree 
that they appear indistinguishable to real persons and can actually be mistaken for such.’ 

One source of this foundational hypothesis was the widespread observation that we 
seem generally willing to accept or are mostly undisturbed by barbarity among artificial 
beings or animated characters in a way that does not hold when they occur between real 
people. Comics and cartoons depict all kinds of violence without shocking us. Indeed, we 
are often amused (as a defensive reaction) by them, even though these same acts would be 
unbearable if we witnessed them being committed by humans. Moreover, it seems that 
animation technology cannot completely or sufficiently model the eyes of artificial faces. 
The gaze, the windows of the “soul” are lacking, or at least are not convincing, because 
the eyes of the avatar are “looking at nothing,” while the gravity of real humans’ eyes con- 


sists in the fact they are always looking at “something” definite. Similarly, the eye and 


1 The research group ended up 
working with FaceRig. We would 
like to thank Holotech Studios 
for kindly giving us permission 
to use the program for research 
purposes. 

2 On the simulation theory, 

the implausibility of which is 
discussed in further detail below, 
see Bostrom (2003). 

— 
3 Onthe phenomenological theory 
of the gaze see Sartre (1992), 

p. 340-400; Lacan (1978), 

p. 67-122; Waldenfels (1999), 

p. 124-147. 


FIG. 1 
PROTOTYPE OF AN AVATAR. of non-human artefacts. The etymological roots of the old German 
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mouth cavities of avatars have no depth. They appear as just another surface receptive to all 
kinds of modifications and projections, such as pseudo-gazes or fake asymmetries between 
the eyes that come from nowhere and, as a result, remain expressionless. ‘Thus, avatars are 
missing authentic facial features. Instead, they confront us with cyberfaces that are at once 
interfaces, which neither see nor conceal anything, but merely achieve visibility by being 
projected onto a screen. 

Other similar intuitive points might help further underscore the plausibility of this 
difference. But they can only serve as guides to help researchers more precisely formu- 
late their goals, questions, and empirical experiments. In this catalog, they provide starting 
points for philosophical, aesthetic, practical, and neuroscientific inquiries into why other 
humans affect us differently than avatars, robots, and the like ever could. We cannot avoid 
identifying with living beings, which might be one reason why their emotional expres- 
sions seem to evoke a different (emotional and affective) reaction in us than even the most 
accurate simulated expressions of artificially generated beings. This observation is all the 
more pressing in an era where the “media sphere” and technologized world cause us to 
interact more and more with quasi-autonomous characters, robots, automatons, and other 
artificial objects. This holds for theater, film, and computer games as well as for things like 
automated chatbots on the internet. The purpose of the interdisciplinary research presented 
in this catalog consists in employing diverse methods and, at the same time, analyzing the 
social, artistic, and cognitive consequences that such interactions have today and might have 
in the future. 

To this end, the researchers intentionally brought concepts and methods normally 
considered incongruent into dialogue with one another. This involved combining approaches 
from phenomenology, cultural anthropology, and media philosophy with the production 
of images, practical settings, and performative experimental constellations at the Zurich 
University of the Arts’ Immersive Arts Space, and, finally, the empirical experiments con- 
ducted with fMRI and EEG monitoring at the Swiss Epilepsy Clinic on subjects with 
and without epilepsy. In order to make the research of all three subprojects manageable 
and compatible, we decided to limit our work to the paradigm of “faciality,” the emotional 
expressions of faces, and how they affect people. We chose as our primary object of study 
avatars as virtual-visual representations of people and, more specifically, avatars’ “faces” and 
their algorithmic production. Thus, the research presented here analyzed all kinds of images 
of faces and humans’ affective and emotional reactions to them. They include sequences 
of images of human faces expressing particular emotions as well as analogous sequences 
of avatars’ faces that, like mirrors, were generated out of the human facial expressions. 
To sharpen our comparative perspective, each series was played through once by profes- 
sional actors, by acting students, and by people with no training in acting. 

However, this choice alone gave rise to complications with terminology. Is it possible 
to talk of “faces” in both cases? Do avatars have “faces” in an authentic sense or just the 

appearance of a face? With the aim of getting around this difficulty, 
the different research projects as well as the catalog distinguish 


between “face” (for the human face) and “visage” for the “faciality” 


» 


signify that which looks at 


134MBODY-AVATAR-MARVELOUSDESIGNER-7701.HTML us, underscoring the significance of the gaze and the moment of an 


unfathomable alterity central to Emmanuel Levinas’s philosophy. 
This alterity is, at the same time, connected to the aura of the human 
face, to its numinosity. In contrast, the duality of “visage” relevant for 
the singular oscillation between faciality and projection surface has more affinity with the 
Latin “facies” and is reminiscent of masks, grimaces, and “making faces.” The terminological 
distinction thus has the advantage of avoiding confusion on the discursive level. ‘This is all 
the more important because the research projects seek to study the potential for confusion 


in perception, particularly when it comes to images. 


AN INTERDISCIPLINARY 
RESEARCH PROJECT 


In this tightknit collaboration, three different disciplines shared their expertise with one 
another in order explore, in both theoretical reflection and empirical experiment, the 
hypothesis about the difference between how humans perceive other humans and how they 
perceive animate artificial artefacts. The philosophical analyses (Dieter Mersch, Jorg Sternagel) 
do not take Bruno Latour’s actor-network theory as their starting point, even though this 
might have seemed like the obvious choice, since it focuses on networks of interlaced or 
overlapping relations. Rather, they work with phenomenological approaches that primarily 
address questions of relating and, in particular, the relation between the face and the visages of 
avatars. Special attention was paid to the role of the human gaze and the difference between 
embodiment and corporeality. Networks are based on the notion of symmetrical or egali- 
tarian relations. However, one of the research group’s primary hypotheses was the manifest 
asymmetry of relations between humans and other humans, on the one hand, and humans 
and artificial beings, on the other, an asymmetry exemplified by the difference between 
intentional looks and artificially animated eyes. Key literature here included the philoso- 
phy of Emmanuel Lévinas, who holds that the experience of the human face maintains an 
exceptional place in the experience of the other; Jean-Paul Sartre’s account of “the look” 
in Being and Nothingness; Jacques Lacan's psychoanalysis of the gaze; and, on embodiment 
and corporeality, the work of Maurice Merleau-Ponty, Bernhard Waldenfels, Kate Meyer- 
Drawe, and Vivian Sobchack. 

Some of the research group’s core questions were: How do we perceive others? 
How do we recognize their “face”? What does it mean to be confronted with a “face” in 
distinction to the “modelled facies” or “visage” of an avatar? And what does it mean to express 
oneself, to make an expression, in contrast to imitating or simulating an expression? What 
are the critical aspects of this difference? Are avatar faces an analog of masks? In turn, how 
are masks different from simulacra? How do we develop a relation to either one? Do avatars 
just have eyes without sight, while the human gaze, as Lacan writes, has to be separated 
from the eye, because it represents an insistence in the imaginary that affects us, stalks us, 
and challenges us, an insistence that we cannot escape? What does the particular gravity 
of the human gaze consist in? From what does its force derive, and can it be artificially 
constructed? Further, are human bodily gestures “expression-laden,” while avatars merely 
produce animated movements? Do they have a body with all of its expressiveness or just the 
formal contours of a corporeal entity? Can embodiment and its fullness be simulated—or 
do avatars and the like simply have similarities, representing a kind of “as-if” body? Does 
Helmut Plessner’s distinction between “being body” and “having corporeality” play a role 
here? Is the innerness of “being body” constitutive for living persons, while artificial beings, 
dolls, avatars, and robots have, at best, a (grafted) exterior body? Finally, what does it really 
mean to maintain a relationship with something or someone different from ourselves? Does 
entering into a relationship always already mean entering into a social relation? Are not 
relationships—in contrast to relations—necessarily situated in a social context, so that we 
can only ever reconstruct them with social and ethical categories? 

One of the group’s primary hypotheses is that relationships of alterity constitute 
a special type of relation that is substantially distinct from relations to artificial beings, 
things, and other objects. Relationships of alterity are rooted in intersubjective acts of 
relating that are defined by concepts like alterity and foreignness, responsivity and pas- 
sibility, witness, trust, and recognition. Something similar can be said of the processes of 
cooperation and communication essential for sociality. These processes do not just consist 
of abstract “inter-actions,” an indifferent exchange of data, information flows, and other 


intentional structures; rather, they are always already founded upon an originary “respon- 


siveness.” Can these forms of relation be transferred to artificial beings without remainder, 
as some theories of post-humanism and media studies claim? Over the course of the research 
projects, it became clear that this problem has serious implications for paradigms significant 
to cultural and media studies, including actor-network theory (Bruno Latour), new mate- 
rialism (Graham Harman, Tim Ingold, Timothy Morton, and others), and technological 
transhumanism, because the research question concerns nothing less than our relation to 
technology, which is constructed by our creativity, but which, as many have noted, has a 
tendency to overpower us and outstrip our ability to maintain control over it. 

A particularly fruitful theoretical model for our inquiry was the “Uncanny Valley” 
(Mori 1970), a fictitious schema developed in 1970 by Japanese roboticist Masahiro Mori. 
According to the graph, artificial beings, animated dolls, robots, and other simulations 
and autonomous actors become more uncanny to us the more “humanlike” or realistic they 
appear. For Mori, uncanniness becomes more pronounced the more the humanoid objects 


are capable of moving on their own or articulating statements. 
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FIGS. 2 AND 3 
SCHEMA FROM MASAHIRO MORI, “THE UNCANNY VALLEY,” AND REPRESENTATION 
OF EMPIRICAL FINDINGS ON INDIVIDUAL PARAMETERS 


FROM MAYA B. MATHUR, FRANCESCA LUNARDI, BALAZS ACZEL, ET AL., “UNCANNY BUT NOT CONFUSING: MULTISITE STUDY 
OF PERCEPTUAL CATEGORY CONFUSION IN THE UNCANNY VALLEY,” DO1I:10.31219/OSF.10/89SF4. 


Of course, the model’s utility requires distinguishing between visual media and “real life,” 
but in general, it suggests that there is a fundamental incommensurability between people, 
on the one hand, and artificial beings, on the other—an incommensurability that cannot 
simply be ignored or circumvented. ‘This necessitates that all-too realistic representations as 
well as fictional encounters with artificial beings in “real life” not transgress certain limits. 
Indeed, according to Mori, it is an ethical imperative of their design that they evidence a 
certain degree of dissimilarity with people. This notion seems especially applicable to artis- 
tic projects in which animated dolls or robots make explicit their “dollness” or “robotness” 
and thus inscribe a moment of distance or reflexivity into the relations that recipients might 
establish with them. 

Nevertheless, it should not be forgotten that Mori’s ruminations on the “Uncanny 
Valley” were heuristic in nature and were not based on empirical study. Since the publication 
of his article, numerous studies in cognitive psychology and the psychology of perception 
have both supported his claims and called them into question. In short, the “Uncanny 


Valley” is just a model, not a proof. Still, the schema is valuable precisely as a heuristic, 


— which, as an unproven assertion, evokes a whole range of questions. The research projects 


detailed in this catalog sought to contribute novel philosophical, artistic, and neuroscientific 
perspectives on this issue. Because if the project’s key philosophical hypothesis that there 
is a hard distinction between humans’ perception of human faces and avatar visages holds, 
it is to be expected that its effects can be observed in the regions of the brain responsible 
for facial perception. Moreover, multiple articles in this catalog engage with Mori’s model 
in order to confirm, criticize, or further develop it and use it as a guide for fleshing out 
the difference between relating to human and avatar faces and analyzing its empirical and 
aesthetic effects. 

The aesthetic-artistic experiments conducted in the mode of artistic research at the Insti- 
tute for the Performing Arts and Film at the Zurich University of the Arts (Anton Rey, 
Gunter Lösel, Miriam Loertscher, Michel Weber) explored this and other issues through 
performative, dialogical studies of facial expressions made by actors and those made by 
avatars either modelled on the actors or directly manipulated by them. Thus, they imple- 
mented in practice what had remained hypothetical in theory. Alongside Faceshift, they 
worked with a number of programs like MSQRD, FaceApp, and Clip2Comic. Particu- 
larly interesting was their work with FaceRig, which enabled actors to play with avatars of 
themselves as if they were acting in front of a mirror and, at the same time, helped them 
establish a practical, reflexive experimental setting that provided optimized images for the 
neuroscientific researchers. The visualizations consisted of parallel images of actors mak- 
ing intense facial expressions for particular emotions like fear or joy alongside the avatar 
“visages” generated from them. These comparable images were then studied as part of the 
neuroscientific experiments. Physiological methods like pulse measurements and eye track- 
ing were used to prepare the images. Their size, proportions, etc. were also standardized and 
digitally refined. Multiple model images were produced and selected for each set in order 
to obtain the best pictures for the subsequent neuroscientific experiments, which sought to 


identify variations in brain activity in reaction to the images. 


FIGS. 4 AND 5 
The production of image ses OWN IMAGES PRODUCED AS PART OF THE PROJECT 


was one part of the goal of cre- FROM INSTITUTE FOR THE PERFORMING ARTS AND FILM, ZURICH UNIVERSITY OF THE ARTS 


ating reflexive self-encounters. 


Another was the performative, 

aesthetic, artistic constellation: actors mimed an emotion, saw themselves in real-time as an 
avatar, and at once controlled their own facial expressions and that of the avatar. For each 
round, they employed clearly distinguishable forms of expression. In a second series of images, 
they then selected the most neutral facial expressions possible. This should not be confused 
with “expressionlessness,” since neutral expressions, too, evoke conscious or unconscious 
reactions from others, not to speak of the fact that neutrality is always indeterminate and 
can hint at a range of different dispositions, such as apathy, ignorance, or masked attraction 
or rejection. On the basis of numerous experimental exercises, the research groups produced 
video recordings and images of people and avatars with varying intensities and densities. 
Then, as if the images were being viewed through a mirror, the actors successively adjusted, 
improved, and made the expressions more precise and expressive. As a secondary effect, the 


procedure could be used to help train acting students improve their repertoire of expressions. 
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FIG. 6 
EXPERIMENTAL FMRI SPACE AT THE Be 
EPILEPSY CENTER AT KLINIK LENGG, ZURICH turn, are concerned with how certain interpretations can 


By engaging with “their own” “facial avatars,” the actors tested out a form of theatre geared 
towards direct examination of experiences of others and of oneself. These experiments were 
then monitored with eye tracking technology and other physiological measurements. 

From the very beginning, however, the uniform definition and typology of emo- 
tions and their physiological correlates in human facial expressions was an object of heated 
debate. The debate primarily concerned Paul Ekman’s theory of the universality of certain 
emotions, their expression, and the ability of people—independent of culture—to recognize 
them. In a number of studies, Ekman identified at least six transcultural basic emotions 
that are embodied in characteristic expressive gestures: anger, joy, sadness, disgust, and 
surprise; an earlier list also contained contempt (Ekman 2007, 2009). But aren’t emotions 
too fleeting to be definable? Is there ever an emotion that isn’t already composed of a number 
of sometimes contradictory emotions? And can emotions be divided up into a vocabulary 
that can then, like a language, be channeled into complex “emotional claims”? Certainly, 
in everyday language, we distinguish between emotions like fear, joy, and melancholy, but 
when we are asked how we feel at a given moment, we often have difficulty pinning it down 
or get wrapped up in contradictions because too many impressions hit us at once. Indeed, 
emotions, which are always rooted in the body, are similar to a microscopic composite 
of antagonistic moods. They can always absorb different nuances or abruptly switch into 
their opposite, so that fear can simultaneously be felt as pleasure and sadness can be asso- 
ciated with the expectation that things are beginning to change. So, what is “an” emotion? 
The one-dimensional register of emotions put forward by Ekman (1972) is just as much of a 
theoretical fiction as the functional iden- 
tification of emotion and facial expression 
undertaken by classical physiognomy and 
its reiteration in algorithmic “automated 
facial emotion recognition” programs 
(Weigel 2020). Emotions are hybrids: 
they can neither be precisely defined nor 
precisely comprehended. They fluctuate 
regularly. And in their simultaneity and 
fullness, they can neither be mastered nor 
totalized. 

Nevertheless, we should not dis- 
miss the fact that humans try to read 
the emotions of others immediately in 
their facial expressions. What a particular 
expression says is less significant than the 


difference between expressivity and neutrality. Actors, in 


be evoked. The experiment’s design was thus intended to 
encompass and work with all these points. The images 
produced in the “avatarization experiments” reflected mimetic opposition. They were 
gradually edited and perfected such that they could be utilized as stimuli for the third 
part of the research project, the empirical neuroscientific studies (Thomas Grunwald, Peter 
Brugger, Hennric Jokeit, Lorena Kegel, Peter Hilfiker, Oona Kohnen, Martin Kurthen, and 
Teresa Sollfrank). The images of human “facial expressions” (actors) and digitally modelled, 
emotionally animated “expressive surfaces” (avatars) were systematically compared such that 
the brain activity triggered by viewing the different visual stimuli might provide data that 
would support the hypothesis that humans relate to the faces of other humans differently 
than they do to artificially generated faces. 

Thus, each of the three disciplines began with situations that, though embedded 
in their fields, were comparable; their material and findings could be placed into relation 


with one another and discussed together. The empirical studies were conducted by means 


of neuroimaging (fMRI) and EEG measurements on subjects with epilepsy and a control 
group without epilepsy. The experiment thus worked with two sets of overlapping vari- 
ables: expressive versus neutral facial expressions and the responses of persons with epilepsy 
and without. The importance of the neuroscientific study should be neither overstated nor 
understated, because it only offers clues or traces, not solid evidence or proof. Still, the 
experiment was designed such that the results of the two groups would make it possible to 
draw inferences about whether epilepsy patients’ reactions to strong emotions (fear) signifi- 
cantly differ from those of people without epilepsy. 

Moreover, the simple, almost intuitive inquiry required that the three groups engage 
in extensive negotiations. They ranged from honing the common terminology—a conditio 
sine qua non for good collaboration—to gaining an understanding of one another’s methods 
to agreeing on how the findings would be interpreted. Because of the project’s interdisci- 
plinary nature, the participants also had to agree on the limits of how the findings would 
be applied and the limits of the different perspectives. The preparation of precise stimuli 
was also a time-intensive undertaking. But in the end, the collaboration’s research question, 


methods, and findings complemented one another very well. 


METHOD, COLLABORATION 
AND TRANSFORMATION 


Interestingly, the collaboration forced the researchers working on the philosophical part of 
the project at the Institute for Theory to expand the scope of their inquiry in various ways. 
Alongside the original question concerning the internal connection between re/ationality 
and alterity (comprised of the genuine ethics, passibility, and responsivity of social relations), 
other relevant aspects of the problem became visible over the course of the research. First 
and foremost was the aforementioned speculative hypothesis about the “uncanny valley” and 
the debates about it among cognitive psychologists that have been ongoing 
since the 2000s. Like the no less fictive “Turing Test” in research on artifi- 
cial intelligence, the “uncanny valley” provides a useful evaluation metric for 
photorealistic visualization strategies in film, computer games, and various 
forms of virtual reality. While a machine’s ability to pass the Turing Test 
(which, up to this point, has never happened) implies a certain indistinguish- 
ability between human and artificial intelligence—a possibility anticipated 
by the spectacular successes of Deep Blue and Alpha Go—the leap over the 
uncanny valley similarly represents a new level of visual illusion. Contempo- 
rary examples of this include deep fakes and the computer generation of faces 
of people who “do not exist,” which are nearly impossible to differentiate 
from real human faces. The dangers summoned up by these technological 
advances, particularly for the social status of truth and falsehood, are clear. 

Another paradigmatic topic were so-called “real dolls.” These life- 
sized, realistic “dolls for adults” offer a material occasion to probe, from 
the vantage of sexual fantasies, the rather abstract question 


about the difference between interhuman social relations 


and human-avatar interactions. Other humanoid “com- FIG.7 


panions” confront us with similar issues, so that we can 
generally speak of a “dollification” of human relations in 
the era of the “technological other.” Dollification denotes 
the reciprocal process in which dolls begin to resemble 
humans while humans begin to resemble dolls, a process 


facilitated by both plastic surgery as well as digital media 
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WHO “DOES NOT EXIST” 


FROM HTTPS://THISPERSONDOESNOTEXIST.XYZ 
IMAGINED BY A GAN (GENERATIVE ADVERSARIAL 
NETWORK) STYLEGAN2 (DECEMBER 2019) 


11 


12 


and their ability to create “hypermimetic designs.” The “technological other” is a “dollified 
person” who has ideal proportions and assumes all the qualities of a “real doll.” This led 
to a series of questions that concerned, on the one hand, both the conceptual distinction 
between dolls and robots—as artificial beings with bodies—and, on the other, between 
masks and avatars—forms of artificial “faciality.” A central position came to be occupied 
by the inquiry into what constitutes a “face,” what it induces, the extent to which it affects 
and challenges us, and the source of the deep significance of the cardinal medium of the 
mask in all cultures. Central, too, were reflections on the realistic presence of faces of people 
who “do not exist.” Are there reliable attributes—too much symmetry, hollow eyes, and the 
like—that can help us distinguish between real faces and avatar surfaces, or is the situation 
more like the legend of Phidias, who created his ideal statues of gods by composing them of 
the most beautiful body parts of different people? On another note, what makes masks so 
unique? What gives them their special ritual energy and symbolic power? 

The research on avatars touched on fundamental questions and conundrums in 
cultural anthropology. The projects sought to distinguish cultural figures like the doppel- 
ganger and literary subjects like that of Théophile Gautier’s novel Avatar from the avatars of 
photorealistic computer games, whose virtuality derives primarily from algorithmic, binary 
constructions (Giinzel 2012, Beil 2012). Ultimately, the analysis sought to reconstruct the 
history and structure of the “avatar,” its uses, and its contemporary resurgence, which pointed 
us back to the notion that the avatar has an autochthonous place in “digital cultures.” While 
this conclusion might stamp the avatar as an artificial product of mathematical-algorithmic 
simulations (Klevjer 2022), it also led to what we might call the “discontinuity thesis” of the 
philosophy of technology, which asserts that “digital figurations,” AI-controlled robots, and 
other humanoid machines represent the dawn of a new era. This era’s artificial beings mark 
a stark departure from their historical predecessors, such as the living statue Galatea, the 
Golem of the Old Testament, Mary Shelley’s Frankenstein, as well as masks, dolls, puppets, 
and the like more generally. The essays in this catalog unfurl some of the consequences of 
these insights. 

The aesthetic-artistic method of the Institute for the Performing Arts and Film at the 
Zurich University of the Arts draws on the techniques of artistic research, which considers 
art itself as a genuine medium of knowledge and knowledge production. Debates about the 
term “artistic research” and its multiple meanings have been ongoing for about thirty years. 
Clear, though, is that it should be seen neither as its own artistic genre nor as a unified 
method. For artistic research, art as such is a mode of research that makes its own singular 
contribution to the production of cultural knowledge, and this without the necessity of an 
interdependent exchange between the arts and sciences. Rather, dialogue between the arts 
and sciences arises only when both sides are granted equal standing and encounter one 
another on their own terms. The aesthetic-artistic part of interdisciplinary collaboration, in 
short, can only succeed when it pursues its own questions and goals. Guided by the same 
initial hypothesis, the research of the Institute for the Performing Arts and Film did not 
just focus on producing adequate, standardized images for the natural scientific part of the 
project. It also engaged with the question of what “expression” means, how it can be embod- 
ied by a face, and how it changes in real time. This helps explain the extensive experiments 
and the numerous prototypes generated through multiple takes, repetitions, adjustments, 
and edits. For their part, the prototypes themselves constitute aesthetic objects of their 
own genre. They functioned like mirrors that revealed how an emotion, whether it be fear 
or joy, can be expressed in a comprehensible fashion by the surface of an artificial figure. In 
turn, repeated engagement with them facilitated the classic work of dramaturgic mimesis. 
‘Thus, the experiment’s set-up did not just consist in an automated real-time translation of 
actors’ facial expressions into avatars. Rather, the ongoing process of improving both the 
actors’ expressions and their avatars helped the researchers and participating actors develop 
a sense for what a certain expression is and says. This necessitated repeated performances and 


evaluations. Subjects—male and female actors—were carefully selected. They rehearsed a 


range of expressions in a range of different situations and received help from a coach. Over 
450 segments and sequences were produced and archived for the purpose of finding fitting 
cuts for the neuroscientific experiments. Simultaneously, the group created aesthetic instal- 
lations that enabled actors to have, in almost therapeutic fashion, an encounter with their 
own avatar. Like theater more generally, this encounter, too, played with transformation. In 
this sense, the avatars enabled a ludification of the self. They opened a dialogue with identi- 
ties whose theatrical aspect was at once a means of self-reflection. 

A series of unexpected hurdles did trip up the group’s work, however. This necessi- 
tated that participants regularly reconstruct and refine the experimental set-up. The produc- 
tion of convincing emotional expressions with a clear meaning with real-time translation 
into avatars was problematic because the uniqueness of the individual actors could not be 
subordinated to standardization and generalization. Moreover, skepticism about the uni- 
versal validity of Ekman’s catalog of basic emotions saw itself confirmed over the run of 
the research. There is no emotional expression that has just one meaning. This necessitated 
the “mirror” situations outlined above, which enabled the actors to reflect on, relate to, and 
observe themselves. The team then got the idea of using similar techniques for rehearsals 
and actor training. One reason was that the confusion of identity and otherness in encoun- 
ters with the avatar version of oneself and its “face” helped actors develop a feeling for how 
they themselves made expressive gestures. Finally, online surveys allowed the researchers 
to collect and analyze data, even though, for reasons of experimental validity, the questions 
were limited to emotions like sadness, joy, and fear, which forced the team to return to 
Ekman’s categories. 

Another complication derived from the fact that expressions can be interpreted 
in multiple ways. The researchers paid particular attention to differences in how recipi- 
ents viewed and were affected by the images of actors and those of the digitally gener- 
ated avatars with their artificial gestures and expressions, which play an important role 
in animated films, computer games, and film presentations. In order to tackle the issue of 
how people relate differently to human faces and avatar visages, the researchers studied the 
limits and possibilities of interactions between living people and avatars by using motion- 
capture technology. This provided a practical test of the initial hypothesis that went beyond 
the experiment with the non-moving images, thus significantly widening its scope. One 
important finding was that movement is an essential component of the affective legibility of 
emotional expressions. Emotions and their communicability are not a function of frozen 
facial expressions, but of animation. Emotional expression is a highly fluid event, a tran- 
sition. The insight led to the hypothesis that avatars—in contrast to the experience of the 
actor’s “face”—derive their attraction primarily from their movements and not from what we 
would call facial expressions, because their surface visage is relatively schematic and does not 
“see anything.” Certain features of the actor’s face like their eyes, eyebrows, and the position 
of both can indeed be modelled without issue. However, their gaze cannot be modelled 
any more than folds and wrinkles can be used to generate an accurate representation of an 
actor’s face. But, then, can we really say that avatars have an expression at all? Sure, their 
features give them “character,” but this character does not have a personality with its own 
history. Some of the contributions in this catalog engage in more detail with this and other 
differences. 

A byproduct of the group’s practical aesthetic research was a series of statements 
about the qualities of artificial beings and their deficiencies, which, despite the advances in 
photorealistic technology, stick to the artefacts and undermine their effectiveness. For their 
part, these insights were highly relevant for the philosophical section of the project. They 
might be provisionally summed up by saying that the difference between human “faces” and 
artificial “surfaces” derives first and foremost from the “holes” that define the face, as Gilles 
Deleuze and Félix Guattari write in A Thousand Plateaus (1987, 167-191). Bodily orifices, 
as negative spaces, play an essential role in facial expressions. Simulating them is difficult, 


because the absence marks the limits of simulation, the point where there is literally nothing 


FIG. 8 
NEUTRAL FACIAL 
EXPRESSION AVATAR 
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Superior temporal sulcus 


to simulate. Thus, it is the “empty” or “unfilled” cavities in human “faces” that, in avatar form, 
simply become black surfaces that mark the places where the inside “turns” outward and 
forge a connection between subjectivity, expression, consciousness, and comprehensibility. 
Because the eye is the focus of attention, just as the mouth is the site of articulation. Both 
gain their significance from the impossibility of representing them, precisely because they 
are not the product of projection or a screen. Rather, they are the product of a depth that 
necessarily remains abyssal. 

Standardized visual representations of the expressions of human “faces” (actors) and 
their avatar counterparts served as the foundation for the empirical, neuroscientific experi- 
ments at the Swiss Epilepsy Clinic. They worked with tools with high spatial (MRI) and 
temporal (EEG) sensitivity in order to detect differences, first, in how people process very 
pronounced facial expressions (of fear) in contrast to neutral facial expressions and, second, 
in how people process human faces in contrast to nonhuman visages. Data was collected 
from two groups: a group of patients with temporal lobe epilepsy (TLE) and a control group 
of subjects without epilepsy. Comparisons between the two data sets aimed at uncovering 
significant influences of TLE on emotional cognition. Guiding the inquiry was the question 
as to whether there were significant differences in emotional perception and the associated 
activation of certain regions of the brain when subjects saw human and nonhuman stimuli 
and when they saw their own face and its avatar version. In other words, the analysis sought 
to disclose whether humans react differently to human “faces” than to simulated expressions 
of the “visages” of artificial beings. One sign of such difference would be the identification of 
discrepancies in the brain activity of subjects with epilepsy and those without. The analysis 
primarily homed in on the degree of difference in reactions to clear facial expressions with a 
very strong emotional or affective dimension. Comparative studies on patients with autism 
and schizophrenia had already suggested that such differences exist. 

The empirical researchers worked closely with those producing the images of actors 
and their digital avatars. Out of 468 possible sequences, they selected 128 video segments 
and then edited them for the purposes of the EEG and fMRI tests. They also put together 
an online questionnaire where 
test subjects could enter their 
assessments of how they expe- 
rienced the intensity of human 
and computer-generated faces. 
Finally, they conducted 56 tests 
with people with and without 
epilepsy. Three lines of ques- 
tioning informed the study. 
First, which regions of the brain 
activities 


indicate differing 


Inferior frontal gyrus 


between the dynamics of human 
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FIGS. 9 AND 10 


RESPONSIVE BRAIN REGIONS IN FMRI. 


FROM THE EPILEPSY CENTER AT KLINIK LENGG, ZURICH. 


emotional expression and those 
simulated by avatars? Second, 
do patients with temporal lobe 
epilepsy show any difficulties in 
perceiving and recognizing the 
visible expressions of humans 
and avatars? And third, what 
consequences should be drawn from this for the design and application of such avatars 
that have different neurological effects on epilepsy patients and the control group? On all 
three questions, both studies (fMRI and EEG) led to promising findings that were pub- 
lished in the peer-reviewed journals Social Cognitive and Affective Neuroscience and Frontiers 
in Neuroscience (Kegel et al. 2020, Sollfrank et al. 2021). This catalog contains a summary of 


the most important results. 


RESULTS AND 
INTERPRETATIONS 


The three groups did indeed work with different perspectives and aims, but the research 
project’s interdisciplinary design with its various methods and fields of expertise forged a 
complex, three-dimensional system that converged in theoretical, aesthetic, and empiri- 
cal insights on the fundamental differences in how people interact with other humans and 
with avatars. Special focus was placed on precisely defining the limits of each subproject’s 
scope and, by extension, on relativizing the reach of the individual contributions, whose 
most fruitful conclusions were born out of their interconnections with one another. Thus, 
the philosophical research of the Institute for Theory, with its emphasis on the aesthetics of 
relationality, initially accomplished little more than refining questions, defining concepts, 
drawing comparisons, and repeatedly hitting on the key difference between “face” (actor) 
and “visage” (avatar). On the theoretical level, this difference—and really, determining this 
was the decisive point—lined up an indissoluble structure of binary relations between human 
and non-human, living and artificial, reality and possibility, and being and appearance or 
“as” and “as if.” The deconstruction of these differences by media and cultural studies only 
touches on the reasoning behind them. However, they remain intact for pragmatic purposes 
and for everyday life. They structure social experience and cannot, despite skeptical analy- 
sis, be simply skipped over. Reducing them to a network of relations and smoothing them 
out into a series of distinctions without difference in which humans and non-humans hold 
the same status constitutes one of the post-humanist misunderstandings that obscure con- 
temporary discourse. Not only are they driven by a manifest confusion of “difference” with 
“dichotomy.” Even more, each dissolution of oppositions leads to obvious contradictions, 
the least of which is the supposed indistinguishability between human and technological 
traits, though the latter are themselves a human construct. Of greater significance, however, 
are hypostatizations of deconstruction. Deconstruction speaks not of factual indifference, 
but, at the most, of the impossibility of grounding strict separations; it only points out 
potential overlaps between the opposed terms, not the inexistence of differences as such. 
That these differences remain indispensable is evidenced by the Zerminological distinction 
between relationship and relation, which was foundational for the entire research project. 
While relationship denotes human connections that always involve interdependencies and 
mutual recognition in a context of structural asymmetries, relations describe formal classifi- 
cations and mappings. Alterity is the primary component of human relationships. The same 
does not hold for confrontations with artificial objects, technological artefacts, and other 
constructs for the simple reason that alterity cannot be programmed. Where relationships 
are equated with relations and thus become exchangeable and substitutable, our conception 
of the social is hollowed out. 

Moreover, avatars are simulated beings without persona and thus without agency. In 
the cases where they seem to have a persona, it is, in the best-case scenario, simulated. 
Simulations are based on algorithms or mathematical functions, in which all decisions are 
determined by preprogrammed formulas. The same holds for their alleged agency. They do 
not act with intentions, but only behave according to binary choices and aleatoric processes. 
Thus, the most they are capable of is generating similarities. But similarity is different from 
equality. In contrast, assuming that human agents are on equal footing with algorithmic 
actions diminishes the reality of the social and reduces humanity to an algorithm. And in so 
doing, it preemptively settles the ongoing debate in philosophy, the social sciences, computer 
science, the study of cognition, and cultural and media studies about human and artificial 
intelligence, on the distinction between thinking and computation and the mathematical 
coding of consciousness in favor of the universality of the digital.* Positing equal encounters 


between humans and avatars, which can only be normal in dystopian science-fiction novels 
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See the critical perspective 
in Betancourt (2020). 
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FIG. 11 


AVATAR: THE WAY OF WATER (2022) 
© DISNEY 


On the opposite position, see 
Manzotii, Smart (2016). 


and films, implicitly accepts the so-called simulation hypothesis, which has been analyzed 
in particular detail by Nick Bostom (2003). The hypothesis holds that it is possible that our 
lives and experiences, our ideas and creations are nothing but simulations generated by some 
supercomputer built by a higher civilization than our own, nullifying the difference between 
“us” and technological artefacts. People like Elon Musk have embraced this idea as having 
a probability of more than one in a billion. However, there are many reasons why it is not 
just unsustainable, but simply absurd, because it not only lacks all sense of the social—its 
genuine koinonia—but also all sense of what consciousness and thinking are, which are 
always consciousness of consciousness and thinking of thinking in the mode of the double 
genitive. This also jettisons the meaning of freedom and action as well as what it means to 
exist as a human with desires, inadequacies, and shortcomings—and thus the need to have 
creativity, art, and philosophy. 

But the simulation hypothesis is also wrong for mathematical reasons, because 
it necessitates not only the simulation of nearly ten billion individual human brains and 
bodies, but also their many interactions with one another, including all their recursive 
simulations and meta-states. The 
sheer calculation of it all would 
exceed the possibilities of the uni- 
verse (the capacity would not only 
demand the simulation of “A,” but 
also all of its potentialities, which 
would propel the necessary actions 
into the exponential).° If we were 
just simulated beings that only 
ever encountered other simulated 
beings, then we would live in a 
completely different world with 
a bunch of reductive descriptions 
and categories. One of the goals 
of the philosophical section of the 
Actor & Avatar project was, to an 
extent, to rescue the indispensability 
of classical philosophical distinctions 
as a countermeasure to their almost 
bizarre dissolution. The idea was 
not to reinstitute the old hierar- 
chical order of metaphysics, but to 
make clear that without them, we 
cannot meaningfully talk about 
ourselves, our own environment, our sociality, and our “community” with technological 
artefacts and other things or creatures. It follows that, as distinctions, they have the status 
of conditiones sine quibus non, not more, but also not less. 

No less mistaken is the “constructivist theory” favored by some researchers in cul- 
tural studies. The theory holds that all perception and cultural life forms are constructed; in 
doing so, it has some affinities with the simulation hypothesis. For its part, though, it is not 
outright wrong, but it is one-sided, because it is forced to maintain the non-constructedness 
of materiality as the condition of all construction. After all, something that is constructed 
has to originate in something else that is itself not constructed. Otherwise, one falls into 
the same kind of recursion problem as with the simulation hypothesis or into an infinite 
regression. Where construction dominates too much, there is nothing that makes construc- 
tion possible. The necessity of presupposing the existence of something non-constructed 
also maintains for intersubjective relationships, which can never entirely be a product of 


relational modelling, but instead are always determined by the relata and where the relation 


stems from. ‘This is the reason for the project’s emphasis on the duality of “relationships” and 
“traits,” which underscore that the social is defined by its passivity. Every construction is the 
production of poiesis and thus of an aczio that collides with the passio of the social, because 
the passive cannot be deduced from the active. 

Alongside such questions of scope and validity, problems of representation and its 
limits also played a central role in the research project. They were explored by using the exam- 
ple of avatars in order to draw broader conclusions, which can be applied not only to visual 
media and strategies, but also to digital, algorithmic, and performative forms of shaping, 
function, and control more generally. The fundamental impossibility of totalizing human 
experience as a social a priori and the limits of its methodological and medial possibilities 
stood front and center. The inquiry into how people encounter and comprehend artificial 
beings, and in particular robots and avatars, served as points of comparison for developing 
insights into mimesis in theater, film, computer games, graphic novels, and anime. 

Some segments of post-humanist discourse and cultural and media studies assert 
that there is no substantial difference between human and non-human agents. The philo- 
sophical section of the current project determined that this claim chronically ignores the 
essential asymmetry and passivity of affects in social relations with others, while unduly val- 
orizing the experience of “other Others” in hyper-mimetic simulations. In contrast, the col- 
laborative research found that artificial beings must be treated as fundamentally different from 
living persons, acknowledging that they can only ever be “similar” to humans, but never 
equal or equivalent to humans. In short, interacting with artificial beings is always defined 
by a deficient similarity. This point is illustrated poignantly by lifelike “real dolls,” which, 
as extreme embodiments of avatars, gradually gained in significance over the course of the 
project. The interdisciplinary research thus rejected the supposed radicality of post-humanist 
theories. Decisive are forms of relationship and their consequences, because social interactions 
between humans are fundamentally different from those between humans and artefacts, 
technologies, and animated beings, since the latter are founded in the former. Decisive, 
thus, is to carefully analyze the specificities of this difference and the reasons behind it. 
This also holds for objects entangled with affects, such as dolls. The heuristic assumption 
of the primacy of social relationships thus occasioned the group to study interhuman forms 
of relating with more nuance than in most work in cultural, media, and art studies (actor- 
network theory, media ecologies, etc.). The real challenge posed by the philosophical part 
of the collaboration consists in the repeated reminder that our relationships with people are 
distinct from those with things and artificial agents, and that social communication (in the 
sense of getting a response) means something different than a mere algorithmic exchange 
with artistic figures modelled on technological operations. The purpose of this insistence 
was not to rehash the anthropological discussion on the chasm between humans and tech- 
nology—this would be pointless, since technology itself is a human construct. Rather, the 
idea was to delineate criteria for the still valid distinction by separating social relationalities 
from other forms of relation. 

Over the run of the research, this project and its conceptual concretization found 
support in the promising findings of the neuroscientific experiments conducted at the Swiss 
Epilepsy Clinic. Further research is certainly necessary, because the experiments were 
focused on a narrow question and could only offer fragmentary evidence. Nevertheless, 
the experiments on strong facial expressions—and particularly those conducted with 
fMRI—came to the by no means obvious positive conclusion that there are both similar- 
ities and differences in the ways that people process human “faces” and avatars’ “visages.” 
It would be going too far to view this as confirmation of the project’s initial hypothesis. Still, 
that the findings do suggest a significant difference is in itself interesting. In comparison 
to neutral facial expressions, those expressing fear made by both humans and avatars more 
strongly activated, in subjects without epilepsy, the inferior occipital gyrus, the ventrome- 
dial prefrontal cortex, the temporal lobe, and the left amygdala (and in the processing of 


human facial expressions the right amygdala, too); in subjects with epilepsy, the activation 
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was less pronounced. For their part, neutral facial expressions performed by both humans 
and avatars elicited roughly similar brain activity, especially in subjects without epilepsy. 
These striking differences between how the perception and processing of human and 
artificial facial expressions activate the brain appeared when subjects were confronted with 
explicit expressions. Differences were especially notable in the anterior and posterior cingu- 
late cortexes, the anterior insula, in the anterior and posterior superior temporal sulcus, and 
in the inferior frontal gyrus. 

Beyond that, the clinical {MRI tests revealed differences in neurophysiological pro- 
cessing of facial expressions in persons with epilepsy and those without. While facial expres- 
sions of fear, in contrast to neutral facial expressions, normally activated the right amygdala 
and the left superior temporal sulcus in the control group, the brain activity of patients with 
temporal lobe epilepsy appeared reduced. The reduced brain activity was identifiable in the 
mesial prefrontal cortex, the superior temporal sulcus, and the inferior frontal gyrus. ‘These 
findings underscore the clinical significance of temporal lobe epilepsy for the processing of 
emotional expressions and provide data that should be considered in future clinical studies 
on how humans process facial expressions made by other humans and by artificial beings. 

In EEG studies of subjects without epilepsy, the event-related potential (ERP) in the 
timeframe of 300 milliseconds after the stimulus presentation demonstrated a significant 
difference between the “visages” of avatars and human faces. In particular, the amplitude of 
negative components was higher at 300 milliseconds after exposure for human expressions 
of fear in contrast with neutral expressions, while the amplitude of positive components was 
higher at 500 milliseconds after exposure for both expressions of fear and neutral expres- 
sions made by avatar “visages.” This suggests that human faces might cause a stronger ini- 
tial arousal than non-human “visages,” while avatars’ “visages” are associated with a more 
delayed, stronger, potentially negative emotional reaction. However, a significant difference 
in responses to facial expressions of fear and neutral facial expressions in both timeframes 
only maintained for human faces, but not for artificial faces. 

The analysis of oscillations in the electrical activity of healthy subjects’ brains deter- 
mined that all faces/visages (neutral and fearful, of humans and of avatars) triggered an 
increase in activity in the form of a desynchronization of oscillations in alpha brain waves. 
Still, all values of responses to the neutral expressions of avatars were significantly greater 
than those for “emotionless” human faces. ‘This might suggest that all stimuli have to be 
processed just because they attract the subject’s attention. Underlying this is the assumption 
that increased synchronization of theta brain waves is associated with an increase in cogni- 
tive processing and memory activities. The data supports the claim that an increase in theta 
wave synchronization, which is correlated with a greater expenditure of resources, is caused 
by facial expressions of fear, regardless of whether they stem from human faces or artifi- 
cial “visages,” while neutral facial expressions do not evoke such synchronization. On this 
particular point, different responses aligned with face type were only observed in response 
to neutral facial expressions. This suggests that processing the impulses given off by avatar 
“faces” requires more resources than with human faces, while facial expressions of fear, 
regardless of their source, might max out the brain’s capacities, so that the different type 
of face carries little weight. In sum, the fMRI and EEG data underscore two points: First, 
there are differences in how our brains process human faces and artificial “visages”—at least 
in the case of strong facial expressions. Second, on the level of methodology, they show that 
the measurements used can generate meaningful insights about how people process human 
faces, artificial “visages,” and the facial expressions of both. This might help future research 
develop better methods to more precisely analyze the effects of epilepsy on emotional and 
social cognition. The studies and their findings can be found in this catalog. 

‘Thus, we are confronted with a striking congruence between the results of the philo- 
sophical inquiry and the neuroscientific empirical research—even if we can only treat these 
congruencies as “clues,” since the cooperation of philosophy and empirical science cannot 


extend further than that. The aesthetic-artistic research of the group at the Institute for the 


Performing Arts and Film opened up another perspective. 
One aspect was its experimental method of creating visual 
stimuli for the neuroscientific studies through repeated takes, 
rehearsals, and recalibrations. ‘This represented a considerable 
effort and more reflexive sensitivity than is normally the 
case with empirical studies, which generally work with 
material that is more or less given. Moreover, together with 
the Immersive Art Spaces of the Zurich University of the 
Arts, the group conducted its own studies on the experimen- 
tal practice of self-encounters. After all, translating their own 
expressions into avatars required the actors to continuously 
“work” on their own faciality—not only on how they made 
certain expressions, such as those of fear, joy, or neutrality, 
but also on their sense of self as such. While the philosophical 
working group identified interactions with human faces in 


contrast with avatar faces as an insurmountable 


difference, the aesthetic-artistic research group 


played with this experience and experimented FIG. 12 - 


with “overcoming” it. Aesthetics and art primar- JOHANNES GUMP P, SELF-PORTRAIT 


ily work not with realities, but with possibilities. 
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This is decisive for the future of artistic practice in 
theater, film, and, more and more, interdisciplin- 
ary art, because these forms’ experimental spaces 
create their own experiential frameworks that 
have always engaged with the problem of encoun- 
tering oneself. 

While artists have historically always used mirrors 
for their self-portraits (see Kruse 2008), today’s digital 
technologies create “other mirrors” that have more to do 
with “doppelgangers” and the free, artistic shaping thereof. 
Virtual encounters with one’s own “face” in an image are 
made possible by techniques like animating or altering it, 
or by motion-capture systems and other three-dimensional 
virtualization programs like those used in film studios. These 
encounters give actors the opportunity to experiment with 
their own identity, to double or distort themselves, in order to 
spur communicative processes that not only reveal the poten- 


tials of avatars, but also make visible the differences in how 


people relate to other humans and to artificial 
beings. These differences, of course, stand at 
the center of the collaborative research project. 
But this section focused less on the nature of 
this difference and its ethical consequences 
and more on art’s “joyful play” with trans- 
formation, which has been part of its practice 
since Dionysian theater. After all, the arche- 
typal god Dionysus originally stood less for 
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intoxication and revelry in pleasure—these were later additions to his meaning—and more 


for metamorphosis and masks. Thus, the inquiry into the use of avatars in digital theater was 


also an inquiry into its relation to the classic mask. It is this desire for “becoming other” that 


generates its own experiences, knowledge, and perhaps even “fear” (see Aristotle), which has 


its own reflexive dimension that exceeds mere pleasure. The social relevance of art and the 


modes of research enabled by it have their own singular, unmistakable power. Beyond the 
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visualizations produced for the neuroscientific experiments, the rich collection of images of 
faces and their various avatarizations represent artificially induced moments of “becoming 
other” that could be studied from other, art-specific vantage points. The current catalog 
documents not just a selection of the images and their experimental use. Multiple contri- 
butions also deal with the aesthetic productivity of the various playful interactions between 


“actors” and “avatars.” 
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Jokeit, Peter Brugger, Martin Kurthen, and Peter Hilfiker, Gunter Lösel, Marius Mahler 
and Oona Kohnen. International partners played a key role in this project. Thank you to: 
Alexander Gerner from the University of Lisbon, Stephan Giinzel from the Technical Uni- 
versity of Berlin, Michael Mayer from the Sinergia group at the Zurich University of the 
Arts, Corinne Soland, Martin Fröhlich and Christian Iseli of the Immersive Arts Space 
at Zurich University of the Arts, Nadja Ben Khelifa of the Freie Universitat Berlin, James 
Tobias at UCR, Allison de Fren at Occidental College, Los Angeles, and photographer 
Elena Dorfman. We were grateful that many other authors participated in the discussion, 
among them Martina Hef ler of the Technical University of Darmstadt, Rune Klevjer of 
the University of Bergen, Kate Meyer-Drawe, professor emeritus at the Ruhr-Universitat 
Bochum and Gabrielle A. Hezekiah of the University of British Columbia. All their 
contributions made possible this rich compendium full of heterogeneous ideas and mate- 
rials, which can be read against the grain or out of order and further developed on their 
own terms or on in conjunction with other approaches. Alongside selected test images and 
figures on their experimental production, it also contains images from art and design, which 
were a source of inspiration for the research groups in different ways. It contains texts, 
essays, and brief studies composed by members of the research groups and by guest authors 
and cooperation partners. Finally, it contains many documents and extracts from literature 
and texts published elsewhere. They all revolve around the research topic in a broad sense. 
‘They offer a range of perspectives on the same subject and open up a discursive field that 
invites readers to continue the work in different ways. Some complement or overlap with one 
another, while others explicitly contradict one another. But above all, they diverge in their 
forms, which range from academic articles, such as the reports on the findings of neuro- 
scientific experiments, to artistic pieces on the “experience of oneself with one’s avatar,” 
which gained significance as a tool of actor training, to essays that stand on their own. 
We thus hope that readers will treat the catalog as a diverse assemblage that is not 
so concerned with delivering complete answers as it is with articulating new questions, 
engaging in reflection, discussion, and experiment, and perhaps spurring other disciplines 


to develop their own inquiries into these issues. 


For all translations from German into English, we are grateful to Laura Radosh. 
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THE AVATAR 


DIETER MERSCH 


THE AVATAR. A LINEAR MATHEMATICAL MODEL 


The original “avatars” were intermediaries or messengers in Indian mythology, comparable 
perhaps to Hermes in Greek mythology. Heralds, they mediated between gods and humans, 
between immortals and mortals, for the latter did not understand the symbols of the former. 
The Hindu term—from Sanskrit, SATIN, avatara—stresses the “descent” from the divine 
to the human, not in the sense of a fall, but in the sense of an emanation or transitionary 
figure. It denotes the transformation of the highest Brahma into the shape of a person or 
animal. The avatara is on the one hand the divine power incarnate in the “soul” of all that 
lives, on the other hand it is the temporal evolution of the divine throughout the order of 
the world and its eras: the transition from the Treta Yuga (silver age) to the Dvarpa Yuga 
(Bronze Age) to the Kali Yuga, the present. Although texts on Vishnu sometimes distin- 
guish between ten avatari stages, ending with Kali Yuga, the great purification, analogous 
to Revelations in Christianity, others see this as a misinterpretation, because all ages must 
be regarded as coequals. 

In his 1856 short story Avatar, an appropriation of the Indian Veden, other holy 
Indian texts, and other obscure doctrines of salvation as is characteristic of nineteenth- 
century European philosophy, Théosphile Gautier tells the fantastic tale of Octave de Saville, 
whose body is slowly being eaten up from within (Gautier 2011). A dialogue begins between 


de Saville and a mysterious healer regarding the illness of the Occident and possible salva- 
tion by the gods of India, whom in Avatar were for the first time accorded a central position 
in European literature. It is also a story of wandering souls, of body switching and stand-ins, 
foreshadowing that which digital avatars of the gaming age in particular are supposed to 
do, as well as virtual assistants on the web that act as mediators between people, machines, 
and information media: the substitution of subjectivity. They are meant to replace the user’s 
corporeality and intentionality with an “all-knowing” (as far as computable functions are 


concerned) simulation that has been freed of all material restrictions. 


In 1992, Neal Stephenson popularized the technical adaptation of the avatar in his science 
fiction novel, Snow Crash. However it is important to distinguish between the mythological 
avataris and its artificial, technological counterpart. While the former acts as a religious 
metaphor for a transformation, the latter are design objects, graphic surfaces, or diagrams 
in a 3-D environment that were created by algorithms. ‘They are completely different from 
classic tools, and also from mathematical surrogates of a persona. They are instead image- 
agents (Bildagenten) that exist in a universe that is entirely iconically defined. Their configu- 
ration equips them with characteristics that follow a logic of decision making that they can, 
within the framework of their program, navigate to hold conversations or chat on certain 
subjects or take on a number of everyday functions that can be useful for the operation 
of computers, technical systems, and internet communication. However, dematerialized 
figures, they can only exist in virtual Euclidean spaces. They also exhibit a strange indif- 
ference to their co-avatars. Asocial at their core, they are completely focused on their user. 
Increasingly, they are equipped with artificial intelligence, advancing to become automata 
able to execute processes on their own and make autonomous decisions regarding specific 
tasks such as translation. 

It is possible, and also plausible, to characterize them as prostheses, in the sense of 
Marshall McLuhan’s theory of extensions, but with the caveat that they do not extend our 
corporeality, but rather our actions and our wills into our monitors (Beil 2012). One can 
also, equally plausibly, see them in the framework of actor-network-theory as actors in a net- 
work of other actors, so that they form 3-D models of artificial Others, which interact with 
us, intervene in our environment, or substitute for us and take over social functions without 
themselves being social. They are like abstract Others without alterity. 

They have been called “virtual bodies,” (Klevjer 2006) “digital doubles,” (Gunkel 
2010) “puppet Homunculi” (Apter 2008) or posthuman “agents” that are beyond any 
differentiation between human and non-human. Like “desiring-machines,” they conjure 
a dialogicity or proximity that is in the process of appropriating our powers of speech and 
expression, of understanding our mimicry and gestures, and in doing so shifting the mean- 


ing of both humanity and sociality.' 


At the same time they are nothing other than images or icons in a limited virtual environ- 
ment in which—it should never be forgotten—everything has been programmed. When we 
speak of “interaction,” it must be remembered that this takes place without exception in the 
imaginary realm. We enter into contact with them as images in images, lending them 
autonomy and meaning, and sometimes even faked identities, like animated characters in a 
film or figures in a novel. Mystifications that see them as fully valid techno-phenomeno- 
logical Others with their own personalities or as elements of a complex techno-social future 
are at best misleading. This is best illustrated by James Cameron’s 2009 movie Avatar, which 
features two kinds of actors, digitalized actors who play “avatars” or people who turned 
themselves into “avatars” with the help of a transformation machine, and artificially ani- 
mated figures. The two different figures illicit different reactions from the audience, in which 


no one, despite 3-D cinematography, believed the illusion that they acted on the same plane. 


1 


On desiring-machines see Deleuze 
and Guattari (1983). See also 
Harroll and Lim (2017). 
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The more important concept behind the design of technological avatars is the creation of 
virtual artifacts that make it easier for us to deal with the opacity of digital environments 
and allow us to move in mathematical environments in a similar manner as we do in our 
daily surroundings. ‘They are therefore programmed to “learn” and “remember” or to “show 
emotions” and hence develop their own “personality,” which is, however, as cybernetic 
as algorithmic storage and self-modification via feedback loops. In order to understand 
their “work,” performance, and range it is necessary to take into consideration that their 
technical/mathematical construction as well as their digital milieu have a purely simulative 
character. Their only perhaps non-simulative element is that they emulate theories about 
human lifeworlds. The scope of their freedom is limited, just as these theories are subject 


to the scientific restrictions of their times, which reveal nothing but historical prejudices. 


Avater “worlds” thus remain in the modus of as if. Their entire reality is appearance and 
similarity, including mimesis or adaptation, so that we are dealing with superficialities based 
on ‘looking like’ that imitate us or provide a historical snapshot of our being, but always miss 
their mark. Avatars, to put it another way, are first and foremost mathematical constructs 
made to engage with us in mathematical spaces that are likenesses of models of how we 
currently view our world. Their precondition is both the model and the modeling of these 
spaces, which have already quasi distorted and deformed the space of the real. In no way 
are these spaces identical with the environment we encounter, and even less so with the 
ambiguity and confusion of social worlds. Rather they are patterns of ordered structures, 
like an algebra placed over myriad fluctuations and spontaneously changing singularities. 
They make an attempt at equality but are more like maps lain over a landscape in an attempt 
to cover it. Even if the maps covered every point of the landscape, they would remain maps, 
for landscapes contain points no more than faces do “landmarks” that can be recognized 
or imitated. For this reason, the outward form of avatars as well as that which they do or 
attempt to express can never be more than a homology. They are imitations that are no more 
than imitations of imitations, because they are images that act in a world of imagery, like 
actors who impersonate a reality that is itself an act, so that it can function as a milieu shared 
by them and human actors. 

Characteristics are faked on both sides: that of the avatars and their expressions and 
actions, and that of the environment, which only approaches human environments when it 
remains closed by a homomorphism. All of avatars’ skills, including their facial expressions, 
are thus artefacts. We are confronted with a completely artificial system that is so concen- 
trated that everything that is possible within it. It is not only part of an anthropomorphic 
game, but is itself already an image, or the projection of a ludification to which the human 
lifeworld has been subject from the beginning. Avatars mimic the perfect mimes in order to 
interact with that which human theories—themselves often no more than phantasms of the 
natural sciences—believe or think about them. That is why there can be no confusing them 
and us. Even AI avatars cannot be perceived or accepted “as” people, not even if they pass 
the Turing test. At most they prove to be the fictions or objects of a techno-imagination that 
invents them in order to embed them in a surface graphic world, just as VR glasses pretend 
to show something “real.” But their adventures remain as foreign to our experience of reality 


as idealized Platonic bodies are to the pyramids of Giza. 


In his philosophical essay On the Marionette Theater, Heinrich von Kleist (1972) examines 
the manikin and its manipulated play. They are simultaneously clumsy as well as moving 
and magical, because the marionettes, dominated by the “wire or string” of the “machinest” 
as the puppeteer is called, lack all self-consciousness. They follow the movement of their 
mechanics, which follows the laws of the center of gravity, so that puppet theater becomes 


a paradoxical “dance,” which, Kleist writes, could not be more perfect because it is antigrav, 


that is, it escapes the inertia of material. For that reason, there is “more grace in the mechan- 
ical manikin than in the structure of the human body,” for gracefulness became “more radiant 
and powerful” as “reflectiveness became dimmer and weaker.” (Kleist 1972: 984, 987). Only 
when knowledge has passed through the absolute and “consciousness” has become infinite 
does gracefulness reappear, so that humans are imperfect beings between manikins and 
God, ensouled only by their attempts at an unachievable perfection (ibid.: 987).? 

This thought can be turned around and used to look at the avatar. Marionettes 
fascinate us because they have the charm of a clumsy, broken, and ungainly corporeality, 
like young children who cannot yet coordinate their movements. They form a caricature 
of human liveliness, whose partial grotesquery is literally deathly earnest, because it points 
towards the finality and fragility of existence and corporeality. In exchange, the figures can 
fly; effortlessly they jump in the hands of the puppeteer, defying gravitation and then sud- 
denly striding with seven league boots to a change of scene. Their mechanisms control them 
like an alien god, so that drill, incapacitation, and the excessive desire for transgression all 
belong together. The marionette, like the mask, plays a game of similarity and dissimilarity, 
whereby the similarity first becomes visible through the dissimilarity, and one reflects the 
other. Its imperfectum is however is for humans perfectum, because its special potential, its 
creative leaps, stem from its imperfection. They are the true poetics of the puppet. 

Digital avatars in contrast neither know anything of such reflections nor of such 
poetics. For them, it is enough to be simulation and surface, which are in turn nothing 
other than the completion of a mathematical formula. An avatar can never be more than 
an approximation, for its image is made out of a network of point-to-point connections and 
their geometrical rotation, as in facial “landmark” detection and in the “motion capture” 
functions used in animation. The closer together the points, the more exact the image. 
But photorealism and hypermimesis never achieve infinity; the forms created can never be 
more than approximations whose strange smooth surfaces—lacking the abrasions of real 
singularities—reveal the failure of identity. Aporetically, the mathematical function—and 
the geometric figure—creates nothing more than an idealization that has no counterpart 
in reality, just as mathematics always comes up against the wall of empiricism. The perfect 
sphere is not a part of reality, and so every mathematical simulation is incommensurable 
with the uniqueness of the living creature. 

Nevertheless, imperfectability is also not unknown to avatars. It consists paradoxi- 
cally in the impossibility of fulfilling their foundational will to perfectibility. The strange 
jolting, angular movements, the uncontrollability of every individual body movement 
alongside facial and vocal expressions—seen in particular in Second Life or when computer 
games change scenes and it is necessary to go from one room into another—result from 
the “logic” of chronic inconsistencies and programming that can perhaps never be perfect, 
because, finite, it can never have enough processing power. But this does not make us recall 
grotesquery, but rather the endpoint of the models, the algorithmic tools, the slowness of the 
processors, and the technical and physical limits of computation. They have no existential 
meaning, only a formal one. They therefore seem to be free of metaphors: no symbolic mean- 
ing appertains to them, at best only an informatical “not-yet”—the never-ending dream of 


the phantasm of mathematical identity. 


Behind the mathematical opacity of the avatar is literally nothing but a hypermimetic 
projection, which can be distinguished by the level of its abstract perfection, but not its 
philosophical meaning. That is why an inexact sketch can be more interesting than the most 
precise copy. Ifa marionette is made of individual limbs with movable parts that are moved 
by an outside power, and masks of cavities and concavities with holes that hide rather than 
reveal, everything on an avatar is like a screen. For avatars possess no gaze and no voice, and 
also no motor function that could express the individuality of their actions. Individuality 


is not an attribute of avatars, at most they have idiosyncratic features. Their dull eyes act as 


2  Translator’s note: There are of 
course many English translations 
of this story, many quite good, 
but none that fit this essay. These, 
like all following uncredited 
translations, are mine. 
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FIG. 2 


THE GEOMETRY OF AN AVATAR MODEL. can discover identities and make 


a screen on which an image is pictured that comes from nowhere and looks at nothing, just 
as their mouths and lips seem to have a secondary sound imprinted on them that does not 
possess a true place and belongs to no one, or as their movements are not situated in and do 
not stem from any body and therefore have neither a center, nor gravity, nor a feeling for the 
environment, nor do they seem to act on purpose. The technological tools are obvious: using 
artificial intelligence and deep fakes, human voices, gazes, and movements are reproduced 
that make use of the unclear relationship between eye and gaze, between mouth and voice, 
and between body, motion, and intention to undermine the sensibilities of the observer 
and purport to be authentic. In this way, a synthetic expression is created, a “gaze” with- 
out shame, interruption, or focus, which cannot search for or observe anything, but looks 
through us, that finds no target, but loses itself. Simultaneously, we are confronted with 
spectral voices with neither source nor addressee, and therefore without the modulation of 
the alterities they are speaking to, for no voice speaks for itself, it is only when it addresses 
an Other. The same is true of movements that do not orient themselves, that neither mean 
something nor complete an action, but operate in sterile, geometrical environments built by 
probability theory. 

And in fact, masks, puppets, effigies, and marionettes have a completely different 
genealogy than 3-D avatars and robots, despite recurrent attempts to interpret avatars as the 
“modern mask.” (Réssler, Schmidt 2010) The two groups are separated by a technical cae- 
sura or disruption between myth and poetics on the one 
side and technology and informatics on the other. The 
former derive from a genuine incompleteness that is their 
fundamental anthropological and existential condition. 
They have a special relationship to death and absence, 
in stark contrast to technological artefacts, which have 
a positive relationship to perfection and infinity and 
bow to a hypermimetic constructivism. Hypermimesis 
does not however mean a heightening of the mimetic, 
for mimesis in Greek is an ethical figure of moderation 
and modesty, since the creatio (physein) is reserved for 
the divine alone. For humans there is only “emulation” 
(mimeisthai); everything else is hybris. Hypermimesis is 
therefore already hubris, and immanent to it is not only 
sacrilege, but also failure. 

The unavoidability of the latter is made clear 
by the utopias of the “as-if” and the idealization of 
appearance, which cannot be fulfilled. They flip the 
relationship of similarity and dissimilarity, manifest in 
the finite. For what is dissimilar is the wellspring out 
of which the similar draws its meaning, provided that 
comparisons and equivalences presuppose the incom- 

parable and difference. It is only 


where we see differences that we 


things become similar (Verähn- 
lichung). Similarity and identity 
are first and foremost reflexive 
concepts. In contrast, simulacra whose calibrations are processes of mathematical models 
target equalities. Their ideal is congruency, equality of structure, and the identification of 
characteristics such as intelligence, creativity, intention, and expression. The entire theater 
of industrial technology, of which digital worlds are in the end no more than a descendent, 
is geared toward reproduction, that is to say toward the copy, both of which are in turn 


derivatives of a logic of representation. Making the absent present in an unending loop, 


doubling the world and reproducing existences: this is the unspoken telos of the nineteenth 
century and of our present era, which seeks to escape death and finality through deceptively 


realistic illusions, but only produces an endless slew of kitsch. 


The desire for the invention and spread of avatars is therefore the exact opposite of our 
cultural desire for masks and dolls, no matter what has been said (Calvillo-Gämez and 
Cairns 2008). They are the mirrors of our lack, which is a condition for our creativity. The 
term “lack,” which denotes missing something, is however misleading, because that which 
seems to be a “lack” is an advantage and a gain. Correspondingly, avatars can never die— 
at most they are deleted—and they remain without tragedy or pain and therefore deserve 
neither empathy nor worry. To the contrary, they are cheap objects of a sado-masochistic 
diversion; embodiment of an unreserved desire for control and power that however, like 
all sado-masochism, tends to hit back, not because of inner resistance, but because of the 
necessary incompleteness of its construction, which also remains unforgiveable. They are the 
true “lack,” which is why we do not shy from taking revenge on them. When that which we 
desire more than anything is fundamentally unfulfillable, then it must be destroyed. That 
is why avatars are not objects that we can love, but that we must hate. Their unfulfillable 
nature is built into the logic of their mathematical design.* Better models with optimized 
resolution or more precise skills might replace their predecessors, but they will invariably 
exhibit the same lack of unattainable completeness. This is in the end true of all artefacts, 
no matter how perfect, for we are always confronted with relative autonomies and the 
impossibility of consumation. 

Furthermore, while ritual masks are part of performative cult and children’s dolls 
belong to a poetics of transformation, the functions of avatars do not even touch the founda- 
tion of the social sphere but stand aside, so to speak, untouched and untouchable. ‘The only 
social function they can fulfil is the substitution of instrumental practices—in caretaking, 
in consulting, as game counterparts, as the help function of human-machine interfaces, etc. 
But as Philipp Stoellger has recently stressed, instrumentality demands reliability, which is 
something quite different than commitment, trust, or responsibility, those underestimated 
foundations of social relationships that constitute its true koinonia (Stoellger 2010). We rely 
on functions, on the smooth workings of operative processes, but we /rust in people, for the 
precondition of trust is “relationship,” not “relations.” That is why, no matter how successful 
the techno-social interactions between humans and machines or their iconic substitutes 
are, there is no human “bond,” in the emphatic sense, or relationship to avatars, merely an 
employment or use. 

That which legitimizes the sociality of the social sphere is the break or discontinuity, 
in one word: mortality. In return, we gain contingency and thus the chronic illegitimacy 
of trust, commitment, and responsibility, which have no norm or rule guaranteeing our 
belief in them making them in every moment vulnerable and subversive, because they force 
revolvere. The technological automatisms and their figurations, of which avatars are only one 
version, seem to make them redundant and to free us from them by replacing the unfore- 
seen and the intrusion of events with ends-and-means operations that are controllable. But 
this implies putting repetition and identity and their fictionalizations at the center of social 
processes and thus privileging infinity, rather than putting difference and finiteness and 


with them time and history in first place. 
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PROSTHETHICS. 
REPLACEMENT 
AND EXCHANGE 


JORG STERNAGEL 


The front cover of a Turkish edition of Théophile Gautier’s 1856 
novella Avatar, originally published in French, bears a reproduction of 
salinocut by the German painter Heinrich Hoerle, which was printed 
in the mid-1920s: it is a “prosthetic head” (Prothesenkopf), which has 
replaced a human head with an artificial one.’ 

The head is portrayed in profile—like a human portrait— but 
at the same time is shown to be a purely mechanical, metallic, and even 
electronic skull, as suggested by visible circuitry around its artificial 
mouth, neck, and occipital area. Only the human outlines imply that 
this is no longer something purely human, but a replacement thereof, 
a sort of prosthesis that stands in for a body part, usually a limb or 
organ. Whereas Hoerle depicts these contours in reference to body 
parts, Gautier portrays them textually in reference to the entire body, 
which is replaced altogether. A physician gives Gautier’s chronically 
ill protagonist Octave de Saville the opportunity to slip into the body 
of another person who, as his “avatar,” not only grants him health, 


but also the affections of 


his previously unattainable 
FIG. 1 love interest. With these 


HEINRICH HOERLE: PROTHESENKOPF, MID-1920S. themes, both Hoerle and 


HTTPS://DE.M.WIKIPEDIA.ORG/WIKI/DATEI:HEINRICH_HOERLE_PROTHESENKOPF.JPG Gautier invite reflection on 
a domain of “prosthethics” 


that arises from a pros- 
thesis as it opens our eyes 
to what can and cannot be 


replaced or exchanged. 


REPLACEMENT 


In one respect, the process of successive replacement unfolds as a situation of conflict (Aus- 


| 


See Gautier (1888). Artwork: einandersetzung, literally “taking-apart”) that has befallen a person, a process that continues 
Heinrich Hoerle in: Backes (1981): 
216 (Druckgrafik no. 31 in the list 
of the artist’s works). many of his other works that address injuries caused either by the recent First World War 


onward via the body. In the case of Hoerle and his print of the prosthetic head—as with 


or by the burgeoning progress of industrialization — the situation is one of suffered violence 
that has necessitated amputating whichever part had been injured or mutilated, then replac- 
ing it with a prosthesis that would allow the person to carry on living, but also working. 
Even if the prosthetic head is more a robotic phantasm of a head replaced with technology, 
thus also replacing human thought, it does indeed belong on the list of body parts that (can) 
no longer serve their purpose, marked by loss and lack. Loss: I am becoming headless. Lack: 
I am missing my head. How can I still perceive, think, and act? What am I suffering through, 
why, and how? Will I be replaced by a machine? What is this “techno-logical” development con- 
fronting me with? 

For Hoerle, this machine is an actual entity of war and of capitalism. The mechani- 
zation of life extends to the body, intervenes in it, mangles it, injures and mutilates it. How- 
ever, it is humans who have authored and accordingly instrumentalized this mechanization. 
Particularly in the twelve lithographs depicting “cripples” in Hoerle’s so-called “Cripple 
Portfolio” (Druckgraphik no. 17) from 1919, he depicts “passionate protest against the 
beastliness of war, an accounting of the enthusiasms and promises of 1914: cripples. the 
madness, that there are people who have been divided in thirds or in half, who go on living 
in an indifferent world; the sickly feverish distortions of crippled existence and their indi- 
vidual forms of expression have been transferred with a quick hand onto these virtuosic 
pages,” as Hans Faber describes it in the 
organ der gruppe progressiver künstler, a bis z 
(official publication of the group of progressive 
artists, a to z), of which Hoerle was the editor 
(See Backes 1981: 116). The same applies to 
his prints Krüppel bettelnd (Cripple Begging) 
(Backes: Ölgemälde no. 18) from 1921 and 
Denkmal der unbekannten Prothesen (Monu- 
ment to the Unknown Prostheses) (Backes: 
Ölgemälde no. 67) from 1930. The figures 
in both works have been given prosthetic 
heads, making visible in art what was already 
visible on the streets: people who had lost 


parts of their bodies during the war, who 


FIG. 2 


HEINRICH HOERLE: DENKMAL UNBEKANNTER PROTHESEN 


(MONUMENT TO UNKNOWN PROSTHESES, 1921). 


HTTPS:/DE.WIKIPEDIA.ORG/WIKI/DENKMAL_DER_UNBEKANNTEN_PROTHESEN#/MEDIA/DATEI: 


HOERLE_DENKMAL_DER_UNBEKANNTEN_PROTHESEN 1930.JPG 


had been injured, who had been casualties of armed violence committed by fellow humans 
(Henninger 2004: 94-96). People who, for example, could no longer reach for something 
with their own hand—because that hand was missing, separated by a hand grenade—or 
who could no longer walk on their own two legs— because one of them was missing, having 
been separated by a false step on a land mine. 

As with Otto Dix’s 1920 painting War Cripples, Hoerle’s artworks also depict 
parts that had already been replaced by prostheses—the missing and therefore invisible 
limbs— exposing a perspective on suffering that is indispensable for understanding what 
makes all violent occurrences violent (their Gewaltsamkeit, or violent-ness), questions with 
which these works confront us. Pascal Delhom remarks: “For the exercise of violence is 


distinguished as such from other types of action only insofar as it is directed against other 
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Above all, this also means not 
simply describing violence using 
terms such as bestiality or animal- 
ity but bringing them back into 
the human realm of responsibility 
in order to emphasize correspond- 
ing relationships clearly. 


living creatures and harms them; insofar as it intends to cause suffering.” (Delhom 2014: 
158) The factor of the exercise of violence, which here is viewed squarely in the context of 
action, involves an inherent premise that this action— that is, the violent occurrence— is 
entangled in its lead-up as well in its lasting aftermath. Violence aims at the face of other people; 
aims at that face’s annihilation, both in its physicality and in its unattainability; and aims at tran- 


scending it by penetrating alterity. 
Lévinas develops this idea as follows: 


Corporeity is the mode of existence of a being whose presence is postponed at the 
very moment of his presence. Such a distension in the tension of the instant can only 
come from an infinite dimension which separates me from the other, both present 
and still to come, a dimension opened by the face of the Other. War can be produced 
only when a being postponing its death is exposed to violence. It can be produced 
only where discourse was possible: discourse subtends war itself. Moreover violence 
does not aim simply at disposing of the other as one disposes of a thing, but, already 
at the limit of murder, it proceeds from unlimited negation. It can aim at only a 
presence itself infinite despite its insertion in the field of my powers. Violence can 
aim only at a face (Lévinas 2007: 225). 


‘The injured war veteran portrayed by Hoerle has already suffered the loss of a body part in 
a war that affected him and other people, and that, as with every use of violence, is attrib- 
utable to human beings. “There is no violence without people who exercise it or who permit 
it to be suffered. And if we do not ascribe a particular instance to a person and nevertheless 
experience it as violence, we can do so because we ascribe it to another entity, to a god or 
personified destiny that we view as an intentional author. Neither a natural disaster nor an 
injurious attack by an animal that humans have not trained accordingly is experienced as a 
form of violence.” (Delhom 2014: 156) Violence is contingent upon human beings. It is part of 
our humanity.’ The same holds for all its applications, which are conceptually described using 
words such as “structural” or “symbolic” because they are not simply natural phenomena, 
and nor do they follow specific causal rules of nature. They are only violent (gewaltsam) 
because they are human-authored, that is, because human beings could have controlled or 
prevented them and their repercussions as natural events or because they, in their produc- 
tion and reproduction, fall within social processes and structures of human responsibility. 
“Thus, phenomena of violence presuppose a causal entity that can act intentionally and can, 
for that reason, be viewed as violent (gewalttatig).” (Delhom 2014: 156) This entity is human 
and cannot be replaced in a phenomenology of suffered violence, so conceived because it holds 
responsibility, especially when the violence results in the injury of other living creatures. 
“For a living creature does not only live in the world, but on its own (von sich aus) has a con- 
nection Zo the world and to the other living creatures in the world. As the subject of this 
connection to the world, it perceives stimuli, discerns objects, moves self-sufficiently, and 
enters into relationships. Hence, it comprises the point of departure and the center of its 
connection to the world and to others. This connection constitutes its life.” (Delhom 2014: 
156-157) At this point, Delhom elucidates that not only does an injury harm the creature’s 
body and its physical constitution, but also has repercussions that include damaging percep- 
tions, movements, and relationships. The injury “affects it in its connection to the world and 
to other creatures. The specificity of the injury lies in this link between the harm to body and 
psyche and the damage to perceptions, movements, and relationships. Only a creature that 
has a connection to the world on its own (von sich aus) can be injured.” (Delhom 2014: 157) 

Delhom highlights that precisely what is affected by a violent occurrence—the 
aspect of on its own, von sich aus—defines the connection to the world and can also be 
described as freedom or spontaneity. “On the one hand, it is the only possible target of vio- 


lence, as an intentional action. After all, what distinguishes it from all other objects is that 


it largely deprives the actor of intentionality and thus interferes with his privileged position 
at the origin and the center of his world. That is why it can only be honored or combated. 
It prompts either recognition or violence. On the other hand, there would be no violence 
either if its chosen target were entirely out of reach.” (Delhom 2014: 157) Violence reaches 
and damages living creatures in their connection to themselves, to the world, and to others. 
It injures them by harming their bodies and their psychological makeup. In this regard, 
Delhom quotes Lévinas: “Violence bears upon only a being both graspable and escaping 
every hold. Without this living contradiction in the being that undergoes violence the 
deployment of violent force would reduce itself to a labor.” (Lévinas 2007: 223) Yet violence 
is not only exercised, it is also suffered, and thus it is first and foremost a misfortune (a Wider- 
‚fahrnis, something “befallen”) and neither an action nor the effect of a structure dependent 
on humans. “The living creature affected experiences it as the Reversal of its connection 
to the world: it does not turn towards the world on its own [von sich aus] but is affected by 
something in the world to which it is exposed. Although as a living creature it is the subject 
of its suffering — it is not the subject in the sense of an author of an intentional act, but in the 
sense that it is subjected (subiectus) to what injures it. The subject of suffering is the subjective 
pole of an occurrence that has befallen it and that it cannot escape. It does not constitute an 
object, but rather it is affected.” (Delhom 2014: 157) This suffering is precisely what distin- 
guishes Gewaltsamkeit, the suffering that takes place in the mutual referentiality between 
suffering and its exercise or causation. “The exercise of violence can, like other forms of 
action, take place as a means to a specific end. It can occur as an expression of hatred, fury, 
or panic; it can be deployed as a demonstration of power and much more. But what consti- 
tutes the Gewaltsamkeit of this means, this expression, or this demonstration is that it is or 
could be suffered.” (Delhom o. ibid. 2014: 158) Nevertheless, Delholm contends that the 
perspective of suffering should not be equated with that of the violence’s victim. Indeed, it 
can also be the perspective of the witnesses of suffered violence, and “it is frequently enough 
the [perspective] of people who exercise or cause violence to happen and are therefore con- 
fronted with others’ suffering due to said violence. Conversely, victims of violence who suffer 
from it can also observe it from the perspective of the act and thus understand or even justify 
it.” (Delhom o. ibid. 2014: 159) “In other words,” Delhom continues, and this is key: “the 
perspective of suffering is not necessarily that of the sufferer. “It also enables observation 
of the entire violent occurrence proceeding from what constitutes its Gewaltsamkeit.”(Ibid.) 

For Delhom, with his phenomenological approach, this is an effort to shift the focus 
to the suffering, a shift based on using terminology that sensitizes to the Gewaltsamkeit of 
violence, which also means clarifying that suffering itself is not an experience in itself: “The 
subject of injury does not constitute that injury as its object, but is affected by the injury. 
Hence, zo the extent that it is suffered, an injury is not a phenomenon. Furthermore, it harms 
or destroys the bodily and psychological basis for experience and movement in the world. 
‘Thus, for the sufferer, it is difficult to address what has been suffered as suffering. Conversely, 
for the other participants or witnesses of the violence, there is always the risk that they 
might overlook the specifically violent [gewaltsam] aspect of suffering within the object 
of their experience, or that they might view the violent occurrence exclusively from the 
perspective of the act.” (Delhom 2014: 160-161) Notwithstanding, in order to scrutinize 
the Gewaltsamkeit of violence as an object of phenomenology and not as either a mere mis- 
fortune (Widerfahrnis)—which is only suffered and not experienced—or as the object of 
utterly constituted experience—which loses sight of suffering — Delhom proposes finding 
an approach that treats suffering as a boundary of experience and adopts a perspective of 
suffering that distinguishes between intrusive and exclusive violence: “Both types of vio- 
lence are suffered as such; both of them affect and injure living creatures in their relationship 
to their habitats, in the first case though the intrusion of a foreign object that cannot be 
integrated and in the second case through exclusion from one’s own habitat. The intrusion 
occurs from the outside in as an injury to integrity; exclusion from the inside out is a denial 


of belonging or integration. Both are injurious.” (Delhom o. ibid. 2014: 161-162) And in 
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3 See, for example, Melanch- 
thon (2018), 6-7. This maxim 
also inspired Cicero, Seneca, 


Montaigne, Herder, Feuerbach, 


Nietzsche, Kant, and Novalis, 
for example. 


both cases, I observe them from within my body, from within my habitat, which I can never 
leave and from which I perceive the world as an exterior world of objects, from which I suffer 


physical violence as an intrusion. 


EXCHANGE 


“No one could understand the disease which was slowly undermining the constitution of 
Octave de Saville. He was not confined to his bed, but led his usual life, and never a com- 
plaint issued from his lips; yet it was obvious that he was dying. He could tell the physicians, 
whom the anxiety of his friends had compelled him to consult, of no particular pain or 
suffering, and their science could discover no alarming symptoms in him. When his chest 
was sounded, the result was favourable; and when the ear was applied to his heart, it could 
scarcely be said that its beating was either too quick or too slow. He had no cough and no 
fever; but his life was evidently gliding away and taking flight through one of those invisible 
crevices of which man, according to the saying of Terence, is full. Sometimes a strange faint- 
ness turned him pale and chill as a statue. For a minute or two he would seem dead; then 
the pendulum of life, stayed for the moment by some mysterious hand, was let go again and 
resumed its oscillations, and Octave woke up as it were from a dream.” (Gautier 1888: 5-6). 

Gautier opens his fantastical novel Avatar with an account of the illness of its main 
character Octave, an illness whose precise characteristics no one seems to know. At the 
start, it remains uncertain, as Gautier describes with a reference to Terence, who speaks of 
“imperceptible crevices in the human fabric.” For Terence, this is something that can be 
affected by another person’s misfortune or unhappiness: “I am a man; I consider nothing 
pertaining to man foreign to myself.” (Terentius 1832: 5 line 25) In an adaptation of a lost 
comedy by the Greek poet Menander, the Roman Terence gives this line to the man who 
punishes himself: the self-tormentor Menedemus who prefers to work in the fields alone, 
without the help of farm hands, who he could afford. He argues that everything human is 
related and therefore also pertains to him and represents a human attitude that includes 
goodness, clemency, and consideration.’ A consequence of humanitas: One must practice temperance. 
The Delphic maxim “Know thyself” also calls for thoughtful consideration (Besonnenheit, 
temperance) of the fact that to be human means to be mortal, which expresses a kind of incom- 


pleteness and limitation, as Plato articulates in the dialogue between Charmides and Sokrates: 


There you are, Socrates, he said: you push your investigation up to the real question 
at issue—in what temperance differs from all the other sciences—but you then pro- 
ceed to seek some resemblance between it and them; whereas there is no such thing, 
for while all the rest of the sciences have something other than themselves as their 
subject, this one alone is a science of the other sciences and of its own self. And of 
this you are far from being unconscious, since in fact, as I believe, you are doing the 
very thing you denied you were doing just now: for you are attempting to refute me, 
without troubling to follow the subject of our discussion. 


How can you think, I said, if my main effort is to refute you, that | do it with any 
other motive than that which would impel me to investigate the meaning of my own 
words—from a fear of carelessly supposing, at any moment, that | knew something 
while | knew it not? And so it is now: that is what | am doing, | tell you. | am exam- 
ining the argument mainly for my own sake, but also, perhaps, for that of my other 
intimates. Or do you not think it is for the common good, almost, of all men, that the 
truth about everything there is should be discovered? 


l Yes indeed, he replied, I do, Socrates. (Plato 1927: 166b-e) 


Temperance in dialogue: The uncertainty of self-knowledge is part of the drama of every body, 
which for Octave takes place in the here and now, as Gautier expressed using words such as 


De 


“faintness,” “strange,” or “mysterious.” Octave is his body, which he Jas as a physical vessel. 
And he is his suffering, which he Aas as an illness. Thus, to follow this main character, with 
Gautier’s help, also means following him in his disturbed relationship to the world, to the 
others, and to himself. Thus the author reveals to us the experience of Octave’s own body, 
his mode of existing. This mode is ambiguous because—to bear with Merleau-Ponty and 
paraphrase him here— if we chose to think of him in the third person as a collection of pro- 
cesses, such as sight, motor functions, and sexuality, we would realize that these processes 
cannot be interlinked and connected to the outer world via causal relationships. Rather, 
“they are all obscurely drawn together and mutually implied in a unique drama. Therefore 
the body is not an object. For the same reason, my awareness of it is not a thought, that is 
to say, I cannot take it to pieces and reform it to make a clear idea.” (Merleau-Ponty 2002: 
231) This ambiguity of the body, its dual existence, prevents Octave, his doctors, and his 
relatives from knowing any more about him beyond the notion that no other road can lead to 
knowledge of him besides that of experiencing, inhabiting (er-leben) his body, “which means 
taking up on my own account the drama which is being played out in it, and losing myself 
in it.” (Ibid.) As such, Octave himself is his body, at least inasmuch as he calls an acqui- 
sition his own. And inversely, his body is like a natural subject, like an preliminary draft 
of his overall existence. “Thus experience of one’s own body runs counter to the reflective 
procedure which detaches subject and object from each other, and which gives us only the 
thought about the body, or the body as an idea, and not the experience of the body or the 
body in reality.” (Ibid.) Here, Merleau-Ponty stresses the ambiguity of the mode of existence 
through the experience of one’s own body, which departs from the Cartesian tradition, in 
which the mode of existence has only two meanings and no more: existence as object and 
existence as consciousness. ‘The object is thoroughly an object; consciousness is thoroughly 
consciousness; the body is the sum of parts without an interior; the mind is itself present 
being without any distance. This is precisely the point at which Gautier begins. Octave tries 
to say farewell to his body’s drama, seeks to leave his sick body as a sum of parts without 
an interior, and takes along his mind as an existence that, to itself, is right there (sich selbst 
gegenwartiges Sein) without any distance. 

Gautier invokes the phantasm of transmigration, which for Octave signifies a way 
out of his existence, which has been determined by illness, but also an absolute fixation 
on another person, on the object of his affection who has been inaccessible to his current, 
impaired body, Countess Praskovia Labinska who is married to Count Olaf Labinski. 
Octave achieves both aims—to escape his body, which is suffering from illness but per- 
haps also from longing, and the possibility of unrestricted access to the countess — through 
an exchange staged in the book. With the help of the physician Balthasar Cherbonneau, 
Octave is able to switch minds with the Count; from that point on, his mind occupies the 
Count’s body: 


Doctor Balthazar Cherbonneau, as he stood between the two inert bodies, looked, 
in his white robes, like the sacrificing priest of one of those sanguinary religions 
which pile up the bodies of men on the altars of their gods. He suggested that priest 
of Vitziliputzili, the hideous Mexican idol of which Henri Heine sings in one of his 
ballads, but his intentions were certainly less murderous. 


He drew close to the still perfectly motionless body of Count Olaf Labinski, and pro- 
nounced the ineffable syllable; which he then rapidly repeated over the profoundly 
comatose form of Octave. The usually odd looking figure of M. Cherbonneau seemed 
at this moment endowed with a weird majesty. The mightiness of the power he 
wielded ennobled his irregular features; and, if anyone could have seen him thus 
performing these mysterious rites with so 
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sacerdotal a solemnity, he would scarcely have recognized in him the Hoffman- 
nesque doctor, who challenged, while he defied, the pencil of the caricaturist. 


Olaf Labinski seemed to be simultaneously shaken with a convulsive agony. Their 
features were contorted, and a slight foam frothed round their lips. The pallor of 
death bleached their skins, while two little quivering bluish gleams, burned with 
uncertain light over their brows. 


Then some strange and wonderful things took place. Octave de Saville and Count 
At a gesture of the doctor, charged to the full with magnetic influence, which seemed 
to point out to them the journey which the master bade them take, the two phospho- 
rescent points set themselves in motion, and leaving behind, as they flew along, a 
track of light, sought their new resting-places. The soul of Octave hid itself 


l within the body of the Count Labinski, and the count’s soul in the body of Octave. 
l The Avatar was accomplished! (Gautier 1888: 56-57) 


From that point on, Octave’s minds lives in the body of another man. This gives him access 
to another world that does not belong to him, one in which he can meet his beloved, as he 
is now also her husband. Gautier has Octave’s mind descend into the body of the count, 
completing a descent that points to the literal meaning of the Sanskrit-derived word avatar. 
In Indian religions, the word avatar refers to the incarnation of a god on earth, especially 
Vishnu. Gautier highlights this cultural etymology early in the novel while setting the scene 
of the physician’s office: 


Along the walls were hung miniature water-colour sketches, the work of some 
painter at Calcutta or Lucknow, representing the nine Avatars already accom- 
plished by Vishnu in the bodies of a fish—a tortoise—a pig—a lion with a human 
head-a Brahmin dwarf-Rama, a hero fighting the thousand-armed giant, 
Cartasuciriargunen—Kitsna, the miraculous child in whom some dreamy minds have 
seen an Indian Christ-Buddha, the worshipper of the great god, Mahadevi-and, 
last of all, he was represented as in deep sleep, in the midst of the Milky Way, lying 
on the five-necked serpent, whose five bent heads formed a canopy over him, await- 
ing the hour of his last incarnation when he should assume the body of a white- 
winged horse, and by letting his shoe drop down upon the universe bring about the 
final destruction of the world. (Gautier 1888: 45-46) 


The Avatar was accomplished. The transmigration has been completed. In Hinduism, an avatar 
is only a vessel and no exchange occurs between the deity and the figure of an animal or 
human; rather, the deity takes animal or human form after “descending” into the avatar. 
By contrast, in Gautier’s version, an exchange takes place in the phantasm of a transmigra- 


tion, which proves deceptive, however, as the following scene illustrates: 


The rest of the breakfast was eaten in silence. Prascovie was not pleased with him 
whom she took to be her husband, while Octave was in a state of mental agony, fear- 
ing lest he might be asked other questions to which he would be quite unable to reply. 


l The countess rose from the table and retired to her own room. 
Octave, left alone, began playing with the handle of a knife, and felt inclined to 


stab himself to the heart, for his position had become almost intolerable. He had 
reckoned upon an immediate victory; and now he found himself lost in the mazy 


labyrinth of another man’s existence, out of which he could find no way. In stealing 
Count Olaf Labinski’s body, he ought also to have possessed himself of his thoughts 
and views; of the languages which he spoke; of his childish recollections; of the 
thousand little private matters which go to the making up of a man’s personality; 
the threads which unite his existence with the existences of others; but for all this 
the skill of Doctor Balthazar Cherbonneau had been insufficient. The misery it was 
to be in this paradise, and yet not to dare to take one single step forward! 

(Gautier 1888: 99) 


Octave’s suffering continues, even as an avatar, because via the actuality of his new body, 
he cannot retrieve what constituted that new body (but also his own, abandoned one): the 
habits, the skills, the linguistic knowledge, the memories, the hidden details described by 
Gautier that make up a person’s selfhood, the relationships that link one person’s life to the 
lives of others. At these moments, Gautier’s prose invites us to perceive Octave’s “avatariza- 
tion,” which has been initiated by suffering and brought about via longings or fantasies, as 
an idea that gradually withdraws from its bodily basis, for in the end the avatar Octave has 
nothing left to say: he cannot access any of the experiences that proceeded his own being- 
right-there (Gegenwart). His being-there is left emptied of sense; he cannot answer to it. From 
there, it is not a stretch to observe the avatar not only in text—via a phantasm, a fiction, 
or something imagined—but also in mediality and pictoriality, in the rise of algorithms 
and digitalization. This demonstrates that the avatar depicted there, with its telepresence 
(Telepräsenz), on the screen, whether controllable or not, having a presence (Anwesenheit) 
without being-right-there (ohne Gegenwart), does indeed create leeway for possibilities of 
self and other between the image and viewer—and shows that meanings can shift accord- 
ingly. Nevertheless, this is certainly nothing more than an ethereal-ization of our world in 
so-called cyberspace, which, as a result of its de-materialization, elevates visibility but does 
not solve the puzzle of invisibility; it exists in a loss of inter-corporeity, and thus cannot 
visibly expose the hidden details mentioned above. As Kate Meyer-Drawe stresses, “No one 
will discover in digitized bodies how humanity will achieve a meaningful world. The visible 
space is becoming dense. It does not leave behind any void in which a Cartesian pineal gland 
could serve as the guardian of sense.” (Meyer-Drawe 1996: 140) Meyer-Drawe builds on 
Merleau-Ponty (2000: 215): “Meaning is invisible, but the invisible is not the contradictory 
of the visible: the visible itself has an invisible inner framework (membrure), and the in- 
visible is the secret counterpart of the visible.”* It thus remains puzzling even if we, joining 
Michel Tibon-Cornillot, assume that an “intermediary” sphere has spread out, “a meta-real 
seam in which the mechanical and artificial takes a place across from its human producers, 
who recognize it for their part as far superior to the classical machine.” After all, according 
to Meyer-Drawe, the meaning “with which we equip our world at their suggestion will not 
be revealed by studying neural networks, nor will it be decrypted by information models. 
To understand that meaning, we have to return to the fact that we are not, after all, glass 
people in a glass world.” (Tibon-Cornillot 1982: 146) In other words, this aesthetics of 
existence would only be perceivable as such if the bodily self becomes tangible as a locus of 
exposure vis-a-vis other people, in obtaining inter-corporeity, in a presence (Anwesenheit) that 
includes being-right-there (Gegenwart), and when corporeity is conceived as the actualiza- 
tion of humanness and as realization and not “avatarization,” because only in realization is 


ineluctable vulnerability and mortality revealed. 


The translation of this contribution is based on chapter 3 in Ethik der Alteritat. Aisthetik 
der Existenz. Wien: Passagen Philosophie, 2020. Permission for use has been granted by the 
publisher. 
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AVATARHOOD 
AND SELFHOOD 


RUNE KLEVJER 


VIDEO GAME JOURNEY 


Avatar” has for more than 20 years been a common term in studies of online games and 
virtual worlds, following its first adaptation into digital culture with the online role-play- 
ing game Habitat (1986) and Neil Stephenson’s novel Show Crash (1991). The concept has 
to a certain extent migrated from online contexts into the study of single-player gaming, 
although “player-character” and similar terms are here still often the preferred term. 

‘There are broadly two different kinds of avatar theory. First, we may study the uses 
and meanings of graphic user representations in digital environments, including player- 
characters in videogames. The job of theory will then be to describe and map out the multiple 
and complex ways in which people interact with and through digital user representations, 
across a broad and diverse field of uses, arenas and social contexts, from Pac-Man to social 
media. 

But we may also apply avatarhood as an analytical concept. Avatarhood is then 
thought of as a version of se/fhood, and a unique principle of interfacing with computer- 


generated environments. Applied analytically to a given context and design, the principle 


of avatarhood is a function, which means that an object or a mechanism is an avatar inso- 
far that it manifests this function. With respect to videogames, this implies that playable 
characters can have other uses and meanings than avatarhood, other functions. 

The concept of avatarhood is useful for identifying and delineating avatar theory as a 
field of study, as well as for analysing notions of “avatar” that are guiding creative practices 
in gaming and digital design. Indeed, in terms of its origins and central position in digital 
culture, the concept of “avatar” is primarily a creative and commercial idea rather than a 


theoretical-analytical construction. 


VIRTUAL BODY \ PERSONA \ 
GAME PIECE 


So what is avatarhood? The first ever graphical online multi-user game, Habitat in 1986, 
was also the first to adopt into digital culture the term “Avatar”, from Sanskrit, which refers 
to the crossing or descent of a Hindi deity (usually Vishnu) into human form, as a material 
incarnation of a deity on earth. Chip Morningstar, one of the game’s creators, felt that avatar 
was an appropriate term in the sense that 
“we humans are like deities, or at least exter- 
nal souls, with respect to a virtual world that 


. . . . . . MA 
exists only inside a computer simulation”!. Lh CELTS 


rou have MA ou 


The concept has since become a popular ref- 
erence for any graphical representation of = Lou haye Fir in 
users or players in computer-generated envi- 
ronments. 

Early literature on avatars shows that 
the general idea of user representation, 
manifestation or “embodiment” in virtual 
environments captures two interrelated but 
distinct ideas. The first idea is that avatar- 
hood is an interface to Virtual 


Reality, a virtual body. The focus 
is en THE FIRST GRAPHICAL ONLINE 


generated environments in the MULTI-USER GAME HABITAT (1986) 


corporeal sense, of the kind 


that is produced by immersive 
VR technology. Frank Biocca’ early formulation of this concept in “Ihe Cyborg’s Dilemma: 


Progressive Embodiment in Virtual Environments” (1997) is instructive: 


In immersive virtual reality systems the avatar is not the small puppet used in 
standard computer interfaces, those regular computer monitors on which an iconic 
representation of the self is moved in a world via a mouse or joystick. In immersive 
VR the whole interface defines the boundaries and shape ofthe body by defining the 
boundary between inside and outside, between the part ofthe VR world that is “me” 
and the part that is “the world” (...) From coherent patterns of energy impinging on 
the senses (i.e., the proximal stimulus) the virtual world is divided into “self” and 


“environment”. (Biocca 1997) 


The idea that avatars are bodies in Virtual Reality has links to the concept of “presence” or 
“telepresence” in VR literature (Rheingold 1991, Steuer 1992). In Biocca’s approach it is the 


embodied self-presence itself that is the avatar (“the whole interface defines the boundaries 


our pocket. # 
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and shape of the body”), not a visually represented character. The idea of avatar as virtual 
body is heavily embedded in avatar etymology; the avatar is a corporeal manifestation of the 
player in a world of ones and zeroes, the player incarnated in digital flesh. 

On the other hand, avatars have more commonly been thought of in expressive and 
interpersonal terms, as virtual personas rather than re-embodiments the corporeal sense. The 
avatar as digital persona is a vehicle of self-presentation, make-believe and social commu- 
nication. Gregory Little’s “Avatar Manifesto” (1999) articulated a variant of this concept of 
avatarhood: “Ihe avatar is a delegate, a tool or instrument allowing an agency to transmit 
signification to a parallel world”. His formulation exemplifies the way in which avatars and 
their “worlds” have typically been conceptualised as a discursive and communicative prac- 
tice. In the field of computer-mediated communication, avatar studies has had a primary 
interest in graphical virtual worlds like Second Life (where user-characters are officially called 
“avatars”) or online role-playing games like for example Star Wars: The Old Republic. Among 
the core concerns in the early literature were the complex and sometimes “deceptive” nature 
of the relationship between offline and online identities (Donath 1999), the ways in which 
the visual language of avatars affects perception and social judgment in virtual communities 
(Nowak, 2004, 2015), or how avatars reflect and propagate stereotypical racial identities 
(Nakamura, 2002). 

The original idea of avatar as persona has also been influential in game studies and 
gaming discourses. Avatarhood is here most often conceived in representational terms, 
as the character of the player, in online worlds as well as in single-player games. At the 
same time, there is an influential tradi- 
tion of theory that de-emphasizes or even 
rejects the significance of player-characters 
in player experience. Fuller and Jenkins 
(1995) argue that Mario or Luigi are not 
really characters in the narrative sense but 
rather “capacities for action”, and as such 
“little more than a cursor”. The observa- 
tion that player-characters are mediators of 
player agency has been developed into the 
idea that avatars are interfaces to a game 
system (Jorgensen, 2013). Such game-entric 
accounts in computer game theory point to 

a medium-independent concept of 
avatarhood. Linderoth (2005) notes 


that “we talk about our game pieces 


STAR WARS: THE OLD REPUBLIC 


in board games as a part of ourself, 

and can in the game of Monopoly 
claim that ‘T stand on chance”. In general game design theory, Bjork & Holopainen (2003) 
define “avatar” as “tokens that represent players”. The notion of avatar as game piece, a 
token representative and mediator of agency within an abstract game system, indicates that 
avatarhood may indeed be given also a formal articulation. One could, after all, play Snakes 
& Ladders or Monopoly without tokens, just as one might do in any game that is not also 


a physical sport. 


AVATAR: THE GENERAL IDEA 


Looking for a general idea of avatarhood that may capture virtual bodies, personas/characters 


as well as game pieces, we may take a cue from Kristine L. Nowak. With reference to 


« 


Snow Crash, she says that avatars “...in some way allow people to be embodied in, and 


have experiences only possible in computer-generated environments” (Nowak 2015:4). This 
formulation connects to the original idea of descent and re-embodiment, while arguably 
being applicable to a range of digital environments, from Pac-Man to Facebook. The key 
idea is that avatarhood is about being “embodied” or situated in a digital environment, “in 
some way”, as opposed to having direct access and agency in relation to a digital space or 
domain, from the outside. 

Hence avatars are not cursors or tools in a purely instrumental sense, but embodied 
representations within computer-generated environments. “Environment” does not neces- 
sarily refer to navigable spaces in VR or videogames, but to any bounded and consistently 
structured domain with characteristic properties and affordances. Via the interface of the 
avatar, we can say that we “descend” upon a computer-generated environment, in a way that 
allows us to relate to it as ovr environment, our “world”, our habitat. This situatedness or 
embeddedness, which I suggest we may call native embodiment, is a defining characteristic 


of avatarhood. 


EXPERIENTIAL AND 
HERMENEUTIC SELFHOOD 


What does it mean that an avatar is an embodiment of the se/f? In the literature on the 
concept of selfhood, there is broad agreement that selfhood can be defined in multiple 
dimensions, so that we can talk of for example “ecological self”, or “social self”, and so on. 
However there are different ways of slicing it up, different approaches to how to understand 
the relationship between different aspects, and different positions with respect to which 
aspects are more fundamental’. 

Dan Zahavi and Shaun Callagher have suggested that aspects of selfhood can 
be grouped under two main dimensions: the experiential self and the narrative self.” The 
experiential self refers to the “I” of subjective experience, and the immediate givenness of 
first-person experience and agency as my experience and my agency. The experiential self 
may also be defined as a “minimal” and irreducible sense of self-awareness; the awareness 
that an experience is mine rather than the experience of another cannot be put into doubt, 
but is a given. Zahavi (2005, 2007) says that ownership of experience is by implication 
perspectival ownership. The self-aware subject experiences in a certain way, in a certain 
perspective; there is, in Zahavi’s words, something “it is like” for a subject to have an 
experience (2007:5). 

In line with classic phenomenology as well as more recent ideas that cognition and 
the self is “embodied” or “situated”, Zahavi argues that the phenomenal self is a priori 
embedded in its environment, even if this relationship may be operating behind our back, 
as it were, insofar that it is not included in our self-awareness in its minimal form. Self- 
experience is “the self experience of a world-immersed self” (Zahavi, 2007:6). This idea 
of the world-situated self draws attention to the self as given in bodily and perceptual 
experience (Gallagher & Zahavi, 2008). It also connects the experiential self to the idea 
of the ecological or “enactive” self, which implies that that selfhood and cognition is con- 
ditioned by the nature of human beings as organisms, “tuned in” to their environment and 
habitats (Neisser, 1988). 

Zahavi and Gallagher’s “narrative” self, on the other hand, is a concept more familiar 
to sociology, literary theory, and media- and cultural theory. The narrative self is the reflex- 
ive and conceptually mediated self, constructed through a work of self-interpretation, and 
can be seen as the expanded or “thick” counterpart to the minimal or “thin” phenome- 
nal self. It is more strongly interpersonal, expanding from the minimal intersubjectivity of 


me-other to the social construction of the self. Significantly, the “thickness” of the narrative 
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See Gallagher (2000) for 
a concise overview. 


The main idea and points of 
discussion are summed up in 
Zahavi (2007) and Belt (2019). 
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MacPherson (2010), quoted in 
Carlson and Taylor (2019) 


self involves an act of individuation, which fleshes out what it is to be “me” in particular, as 
opposed to someone else. In contrast, the experiential self is a “dry” or formal category, an 
attempt to articulate selfhood as such, in its most abstract and minimal sense. 

The narrative dimension refers to the active shaping of the self, or in a stronger term, 
the self as an ongoing project. In a highly pluralistic and differentiated society, the her- 
meneutic self-project will typically be situated not only in a general historic and cultural 
context, but differentiated as multiple ro/es manifested in relation to different situations and 
contexts. Self-projects and roles are often made socially explicit through signifying and 
discursive practices, and through cultural activities of self-expression and self-authorship. 

Avatarhood, then, may productively be seen as a proxy version of the kind of selfhood 
conceptualised by Gallagher and Zahavi. To avoid confusion in the context of avatars in 
digital media, Aermeneutic is probably a better term than «narrative». In the hermeneutical 
dimension, selfhood is understood as a process of reflective self-interpretation and self- 


construction. 


DIGITAL PROXY SELFHOOD 


In general terms, a “proxy agent” is an authorised mediator, a representative who acts on 
behalf of an outside actor, an authorizer, in a given situation.* A proxy agent mediates 
ownership of actions, in such a way that the actions of the proxy agent count as the actions 
of the authorizer, who is then also responsible for their consequences. We may say that an 
avatar similarly functions as a proxy of the user or player. When I am playing a videogame 
via an avatar, the failure of my avatar is my failure, and the weakening of my avatar within 
its environment is my weakening in the environment. At the same time, the notion of proxy 
selfhood, inspired by Gallagher and Zahavi, implies a relation that goes beyond agency and 
representation in the instrumental sense. 

Proxy selfhood is defined by two proxy relations: 

Proxy identity, or se/f-embodiment, means that avatars are embodied externalisations 
of the self, mediating a subject-position and experiential ownership. Experiences via digital 
proxy embodiment are my experiences, from my perspective or “point of view”. There is 
something “it is like” to be me as a particular avatar. Proxy selfhood is a monadic relation, a 
self-to-self relation. This distinguishes avatarhood from the relation between an authorizer 
and proxy agent, which is a dyadic relationship. It also differentiates avatars from the kind 
of dyadic relationships we are familiar with from literature and cinema, in which we are able 
to identify with or project ourselves into another person, imagined or real. Avatarhood is an 
identity relation, not a relationship of identification. 

Proxy situatedness, or native embodiment, is the avatarial version of situated or embed- 
ded selfhood. The term “native” highlights avatarhood as interface, a traversal function, a 
re-embodiment of the self across the human-digital divide. Unlike a purely instrumen- 
tal interface (like mouse cursor or voice command), digital proxy embodiment situates the 
player or user as a natural member of the environment. In native embodiment, the “passive” 
or centripetal relation between avatar and its environment —the ways in which agents and 
forces can affect upon it— is as important as the active or outward-moving, centrifugal rela- 
tion of agency and mastery. ‘The centripetal relation is of particular importance in gaming. 
Via the principle of avatarhood, video game players are being subjected and exposed to an 
alien and often hostile environment, as native inhabitants. 

Native embodiment is an alterity relation, a temporary re-embodiment of the self 
as a technological construction, the self as nonself. In cultural and psychological terms, 
participation and engagement through avatarial interfaces thrives on this underlying 
paradox. In avatarhood, selfhood is being offset and put into play; I am experiencing a 


version of my self that is native to an alien environment. 


Native embodiment is also a ro/e at the level of experiential selfhood, a pre-reflective 
externalisation of the self, which may be brought to awareness and fleshed out in the 
hermeneutical dimension. This role is also very often designed to be an active proxy rela- 
tionship, a project of self-construction and self-authorship, via the technological tools of 


avatar configuration. 


DOMAINS OF AVATARHOOD 


The concept of avatarhood as outlined above is rooted in video gaming and virtual worlds, 
but can also be applied more widely to throw light on avatar-like aspects of digital identi- 
ties online and in networked society in general. With respect to gaming, the avatarhood 
approach emphasises situatedness and the centripetal dimension more strongly than estab- 
lished accounts in game studies, which overall tend to focus their attention on agency and 
creativity. The idea that avatarhood is a proxy self-world relationship also points to the 
existential relevance of avatars, as has been examined by Vella (2015) and Kania (2017). 

With respect to online sociality in virtual worlds, attention to the experiential 
dimension of selfhood draws attention to the way in which avatars are corporeal exten- 
sions as much as communicative ones, even if this aspect tends to be given less attention 
in psychological and sociological research. In their study on avatar personalisation in three 
virtual worlds Ducheneaut et.al (2009) concluded that “...avatars might be a better vehicle 
to explore new forms of physical embodiment rather than for exploring new facets of one’s 
personality (1160)”. 

The notions of proxy selfhood and native embodiment go against the popular idea 
of Total Immersion, which has been seen as the holy grail in a long tradition of immersive 
amusement technologies, and which has also 
been reflected in Hollywood fantasies like 
The Matrix or Avatar. This VR ideal posits 
that virtual embodiment should be, as much 
as possible, a replica of natural embodiment, 
and that, accordingly, interfaces should be 
naturally immersive and immediate. In 
contrast, the principle of proxy selfhood 
accentuates the alterity dimension of digi- 
tal self-embodiment, which is premised on 
dedicated learning and habituation rather 
than sensorial immediacy. 

Journey (Thatgamecompany, 2012) 
illustrates the significance of corporeal 
virtual embodiment and learned proxy 
embodiment in online environments. In this game there is no verbal communication 
when player-avatars meet and join, no identity markers, and limited possibilities for visual 
expression. Indeed there is very little beyond the experience of being physically together 
in a particular place, by extended proxy. This intuitive sense of bodily co-presence is not 
produced via immersive VR interfaces, but via the learned motor-perceptual practice of 
inhabiting the lands of Journey as avatar, via the fingertip interface of the game controller. 

With respect to identities in social media, proxy selfhood is anchored not so much in 
the visual representations of users (which may sometimes be referred to as “avatars”), but in 
the activities and characteristics of a user profile as such. As in Second Life and similar virtual 
environments, saying that your social self on Facebook is simply yourself in front of the 
computer is not the whole story. In this context, the concept of avatarhood serves to high- 


light not only the mechanisms of active self-interpretation and self-presentation involved 
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in maintaining our online social personas, but also the formal definition and positioning of 
ourselves as avatars in an algorithmically structured social and public environment. Such 
formally articulated avatars, like any avatar, are only in part constructions of our own design. 

As a special variant of “extended selfhood” (Belk, 1988), the concept of avatarhood 
can also be applied more broadly, to address native embodiment in the dataspaces of modern 
life. Avatarhood in this expanded sense would apply to the “you” that is being targeted by 
google ads online, as well as to our digital identities as defined by government, work and 
career. E-government platforms, workplace portals, and other personalised interfaces make 
formal identities accessible and manageable in a way that was unimaginable just thirty years 
ago. This development arguably changes the nature of formal identity from public record to 
proxy selfhood. As in the games we play, our avatarial selves in digital society are vehicles 
of self-monitoring and scorekeeping, in ways that resonate with broader trends towards the 


gamification of modern life. 
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THE SELF AND 


ITS AVATARS 


IN NEURO- 


PSYCHIATRY 


PETER BRUGGER 


Musing about the similarities between reduplications of the self in neuropsychiatric 
disorders and personalized avatars in virtual reality (VR) video games invites questions 
fundamental to both cognitive neuroscience and cognetics (BOX): What are the conditions 
for embodiment? Is self-duplication possible? Can a “disembodied state” be induced? How 
can somebody made feel at a remote place, inhabiting another body, acting at a distance? 
What role plays perspective taking in real and virtual environments and how important are 
individual differences in the ability to switch perspectives? What if one’s avatar looks like 
oneself, moves like oneself, feels like another self—yet, seeks to dominate oneself? Is there 
a thing like “thought insertion” by an avatar? Can avatars promote altruism? What makes 
them powerfull marketing agents? 

The present essay is an introduction to the complex interactions between the self 
and some “natural avatars”, i.e. variants of the self that may be experienced in neuropsy- 
chiatric conditions. It offers a systematic classification of autoscopic phenomena (BOX), 
i.e. of doppelganger’ as they manifest themselves with or without overt brain damage. The 
classification is based on phenomenology (Brugger et al., 1997); doppelganger can be seen, 
merely be felt at a specific location in peripersonal space, or be experienced as a multimodal 
mixture of seeing and feeling. Neurological avatars, brain-generated reduplications of the 
self, may develop a will of their own and at times even oppose their owner’s intentions. In 
fact, hostile interactions between a person and her antagonistic doppelganger are among the 
most astounding distortions of self-experience. The belletristic literature is a rich source of 
descriptions (Dieguez, 2013)? and reflects the broad range of doppelgänger manifestations. 
I review the clinical literature by beginning with the least dramatic kind of doppelganger, 


the mere visual reduplication of the self. 


THE VISUAL DOPPELGANGER: 
AUTOSCOPIC HALLUCINATIONS 


In an autoscopic hallucination, a patient sees an image of himself “as in a mirror”. In the 
old French medical literature, these visual reduplications were consequently referred to as 


“hallucinations spéculaires”, i.e. mirror hallucinations. Colors may be vivid and the focus is 


| 


The English language literature 
usually writes “doppelganger”, 
avoiding the umlaut “a”. | use the 
German spelling (though with a 
small-cap “d”), which does not 
change the English pronunciation. 
Note that the German word has 
no “s” in plural; it is one doppel- 
gänger and two doppelganger. 


The anthropological literature 
on doppelganger phenomena 
is revealing as well. A classic is 
Rank (1914), whose psychoana- 
lytic stance may be apologized, 
but whose treatment of the 
universal use of symbolic forms 
of reduplication (comprising 
the soul, shadows, reflections, 
self-portraits, twins and dolls) 
is impressive both in scope 

and detail. See also Todd and 
Dewhurst (1962). 
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FIG. 1 
IN AN AUTOSCOPIC 
HALLUCINATION, 
ONLY VISUAL FEA- 
TURES ARE REDUPLICATED AND THE 
DOPPELGANGER IS A MIRROR IMAGE. 
BRAIN LESIONS TYPICALLY INVOLVE 


THE VISUAL CORTEX, MORE FREQUENTLY OF THE RIGHT HEMISPHERE. 
LEFT: A PATIENT WITH A BILATERAL OCCIPITAL STROKE SAW HER DOPPEL 
GANGER MIMICKING HER GESTURES. SHE NOTICED THAT MOVEMENTS 
WITH HER LEFT ARM MADE THE MIRROR DOUBLE MOVE ITS RIGHT. THE 
HALLUCINATED SCENE PARTIALLY COVERED OBJECTS AND PERSONS IN 


THE BACKGROUND 


(FROM ZAMBONI ET AL., 2005, FIG. 2). RIGHT: ADOPTED FROM BLANKE AND METZINGER, 2009, FIG. 2. 


w 


Le Horla is the one “out there” 
(hors-de-la), which haunts Guy 
de Maupassant’s protagonist 
in a 1886 novel. He describes 
in gripping diary style how the 


invisible being drives him mad. 


In the end, the protagonist 
decides to kill the sensed 
presence, only to recognize in 
the concluding sentence that, 
to effectively get rid of it, he 
will have to kill himself. 


on visual detail. Autoscopic hallucinations are always pathological and indicative of some 
damage to the “visual brain”, i.e. its most posterior parts (Fig. 1, right panel). Right-sided 
lesions are more frequent than left-sided lesions, in accordance with the right hemisphere’s 
stronger involvement in face processing, in particular in the processing of one’s own face 
(Keenan et al., 2001). The duration of an autoscopic hallucination is variable, but tends to be 


short, especially when it occurs as a seizure manifestation. 


THE SENSED PRESENCE: AN 
INVISIBLE DOPPELGANGER 


The convincing awareness that somebody else is nearby has the same perceptual quality as a 
phantom limb, whose presence is only felt, vividly and precise with respect to spatial loca- 
lization even in the absence of any visual confirmation. Although sensed presences are not 
always recognized as one’s doppelganger, an almost eery feeling of belonging evokes strong 
bonds with “the one out there”’. Identification as a second self occurs when the presence 
imitates a patient’s bodily movements. Sensed presences have been conceived as “coenes- 
thetic doppelganger” (BOX coenesthesia), that is, duplications of the bodily sense, without 
any visual component. Accordingly, lesion location in cases of neurological patients is more 
anterior to the visual brain, i.e. mostly in the parietal lobes (Critchley, 1955) or in the region 
of the insula (Heydrich and Blanke, 2013; see Fig. 2), which processes nonvisual aspects of 
inner bodily functions. The sense of presence is also experienced by healthy individuals; it 
occurs especially frequent in single-hand sailors or mountaineers, when they are exhausted, 


dehydrated, socially isolated and in immediate danger (Suedfeld and Mocellin, 1987). 


Patient's left temporal lobe tumor 
originating in the insular region 


FIG. 2 

A “SENSED PRESENCE” (ARROW) CANNOT BE DIRECTLY VISUALIZED. NEVERTHE- 
LESS, A PERSON CAN LOCALIZE HER INVISIBLE DOPPELGANGER IN THE SUR- 
ROUNDING SPACE AS ACCURATELY AS AN ARM AMPUTEE CAN LOCALIZE HIS 
PHANTOM HAND. CORTICAL CORRELATES COMPRISE LESIONS TO THE PARIETAL 
OR TEMPORAL LOBES, BUT THE PHENOMENON IS FREQUENTLY EXPERIENCED BY 
NEUROLOGICALLY HEALTHY PERSONS. THE FIGURE DEPICTS AN INSTANCE OF A 
STRICTLY RIGHT-SIDED “POLIOPIC HEAUTOSCOPY” EXPERIENCED BY A MAN IN 
HIS EARLY FORTIES WHO, AS THE FIRST MANIFESTATION OF A TUMOR IN HIS LEFT 
TEMPORAL LOBE (RIGHT PANEL, ADOPTED FROM HEYDRICH & BLANKE, 2013, FIG. 1H), 

EXPERIENCED AN ENTIRE “DOPPELGANGER FAMILY”, WHOSE MEMBERS KEPT A FIX 
DISTANCE TO HIS BODY WITH A DECREASING PSYCHOLOGICAL AFFINITY WITH 
INCREASING DISTANCE. THE SON WAS AN INVISIBLE DOPPELGANGER, NEVER 
VISUALIZED, BUT DISTINCTLY FELT AT A DISTANCE OF 20 M 


(ADOPTED FROM BRUGGER ET AL., 2006, FIG.2). 


HEAUTOSCOPY: THE CLASSIC 
DOPPELGANGER EXPERIENCE 


The term “heautoscopy” alludes to the self being split into a perceiving and a perceived part. 
In contrast to an autoscopic hallucination, a person does not see a mirror image of herself, 
but another self. Visual features are of secondary importance, the doppelganger can look 
very differently, represent an image of oneself from a different aget, or of the opposite sex 
(“heterosexual heautoscopy”, Fig. 2). The defining feature of heautoscopy is the feeling that 
the seen figure is another self, a doppelganger. Selflocation can be ambivalent; at one 
instance the person appears to look at her doppelganger, at another instance she may Ze the 
doppelganger who looks back onto her body (see the next section). It is this existential 


uncertainty about who and where one really is that leads to a bewildering variety of most 


> 


Most frequent (and terrifying) is 
to A A . ane a pronounced age-progression of 
this is the reason for the fact that the motif of the double is abundant in belletristic treat- the doppelganger (Galant, 1929), 


complex cognitive and emotional interactions between self and doppelganger. No doubt, 


ments of neuropsychiatric signs (Dieguez, 2013). While beneficial interactions in the form culminating in seeing it dying or 
attending its funeral (Lhermitte, 


of the doppelganger taking the role of a guardian angel or of a consoler are described, hostile 1951). 
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interaction predominate both the belletristic and clinical literature (Brugger, 2007). 
Many writers describing doppelganger phenomena in their work have reportedly experi- 
enced heautoscopy themselves. Probably the most famous example is Fjodor Dostoevsky 
(“The Double”), whose autoscopic experiences were triggered by temporal lobe epilepsy 
(Menninger-Lerchenthal, 1946), but also Johann Wolfgang von Goethe gave an auto- 
biographic account of heautoscopy, which he experienced, neurologically healthy, in a criti- 
cal moment of his life (Menninger-Lerchenthal, 1932). In clinical practice, the theme of 
“heautoscopic suicide” illustrates that the literal killing of oneself may be avoided by having 
one’s doppelgänger commit suicide (Arenz, 2001) or by attempting to kill one’s virtual 
rather that one’s real self (Lukianowicz, 1958; Ames, 1984). These symbolic ways of extreme 
self-harm have also been famously described in the work of renowned writers. In “William 
Wilson”, Edgar Allen Poe has the hero stab his doppelganger and Oscar Wilde (“The por- 


trait of Dorian Gray”) describes how the protagonist, who intends to attack his self-portrait, 


FIG. 3 

THE DOPPELGANGER IN THE MULTISEN- 
SORY EXPERIENCE OF HEAUTOSCOPY IS 
NOT A MERE MIRROR IMAGE OF ONESELF. 
HIS HANDEDNESS IS PRESERVED, NOT 
MIRROR-REVERSED. IT IS SEEN AND FELT, 
MAY TALK TO THE PERSON, OR SHARE 
THOUGHTS TELEPATHICALLY?. ITS SUDDEN 


APPEARANCE IS OFTEN FRIGHTENING (LEFT PANEL FROM BANGS, 1898, P. 116). IF EXPE- 
RIENCED IN THE COURSE OF BRAIN DAMAGE, THE LESIONS GRAVITATE TO 


THE LEFT TEMPORAL LOBE (THE RIGHT PANEL, ADOPTED FROM BLANKE AND METZINGER, 2009, 
FIG. 2, DEPICTS A LESION OVERLAY OF SEVERAL CASES). 


— 
Apparent telepathy is often 
experienced in autoscopic 
encounters. It is as virtual as the 
doppelganger itself; extrasensory 
perception does not exist in the 
real, physical world (Brugger and 
Taylor, 2003). 
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is found with a knife plunged in his heart. Assaults by one’s own doppelgänger have also 
been described, both in literary accounts and medical case reports. Thus, the hero in 
Dostoevsky’s “The double” is first humiliated, then physically attacked by his doppelganger. 
A patient of Bakker and Murphy’s (1964) was found with her tongue cut. She denied self- 
injury, but insisted that her aggressive doppelganger had attempted to strangle her and 
attacked her with a knife. Less dramatic appear fictious accounts of persons, who kill them- 
selves in order to escape the burden imposed by heautoscopy; Rudyard Kipling (“At the end 
of the passage”) and Friedrich von Gerstäcker (“Der Doppelganger”) each provide an exam- 


ple. However, the same motif found in medical records may provide a most dramatic read. 


Arthur Wigan, an early scholar investigating psychiatric aspects of functional hemispheric 
specialization, reports the case of a gentleman, who day and night felt persecuted by his 
doppelganger. He finally shot himself to stop the horror (Wigan, 1844/1985, p. 96). We 
described an epileptic patient’s fenestration in an attempt to “stop the intolerable feeling of 
being divided in two” (Brugger et al., 1994, p. 839). 

Identification with the doppelganger in heautoscopy is greatly enhanced over identi- 
fication with one’s mirror image in an autoscopic hallucination. This is because in the latter 
form of reduplication only the visual features are mirrored, and they are mirrored in a literal 
sense: the autoscopic doppelganger waves its left hand if the person waves her right (Fig. 1). 
In contrast, one’s heautoscopic double has always the same handedness as its owner; not only 
visual features are projected into the image, but also body schema which represents non-vi- 
sual bodily sensations such as size, heaviness, interoceptive feelings and motor asymmetries, 
including hand preference. Awareness of the sum of these nonvisual properties of one’s own 
body is sometimes designated as “coenesthesia” (BOX). It is these non-visual components 
that bring about the uncanny dimension of autoscopic phenomena, including ideas of perse- 


cution and the urge to kill the doppelganger (see also footnote 3). 


THE OUT-OF-BODY 
EXPERIENCE: FROM SEEING 
TO BEING ONE’S DOPPEL- 
GANGER 


Investigations of out-of-body experiences (OBEs) were long frowned upon in the established 
medical and psychological sciences. They belonged to the domain of “psychical research” 
or parapsychology. These fields provide valuable descriptive accounts of the phenomenon, 
but have added little to an understanding of the underlying mechanisms. Naturally occur- 
ring OBEs give testimony to the tight interplay between emotional and spatial processing. 
In situations of life-threatening danger, the person feels detached in a double sense: the 
(apparent) physical detachment from the body is accompanied by a corresponding emotional 
detachment. Thus, during a fall, a mountaineer may watch a body fall, discover with amaze- 
ment that it wears the same cloth and an identical rucksack as he does, and matter-of-factly 
but emotionally unconcerned, conclude that this must be himself (Brugger et al., 1999; 
Blanke and Dieguez, 2009). This protective function of an OBE is particularly well illus- 
trated by accounts from those who were convinced to face death, but the double-detachment 
is a more general feature of autoscopic phenomena, specifically heautoscopy. When death is 
not immediately imminent, the doppelganger may still act as a consoler. ‘Thus patient often 
ascribe their suffering to the one nearby and ask the medical personnel to first care about 
their doppelganger, as it would be him, who actually was in great pain. Even more prosaic 
is the instance of the doppelgänger, who did the lawn-mowing a patient was expected to 
do, but felt too fatigued and unmotivated (Devinsky et al., 1994, case 9). The concept of 
transitivism (Wernicke, 1900), i.e. the projection of own symptoms, intentions or actions 
onto others, is crucial here; some scholars devoted entire monographies to this transitivistic 
role of autoscopic phenomena (e.g., Müller-Erzbach, 1951; Mikorey, 1952). Transitivistic 
projections come in degrees: while unilateral somatic signs are usually projected onto other 
people in a mirror-like fashion (Gloning et al., 1957), they respect handedness when pro- 
jected onto one’s own doppelganger, i.e. a left-sided hemiparesis manifests as such also on 
the doppelganger (Brugger, 2002°). In OBEs, protective split of the self into two selves just 
goes that far, that a person will rather be than see a doppelganger. It is this complete illusory 


projection of the bodily self (BOX) into extracorporeal space, which consoles an individual 


oO 


This reference for an extensive 
discussion of the joint role 

of somatic and psychological 
perspective-taking. 
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FIG. 4 

IN AN OUT-OF-BODY EXPERIENCE THE OBSERVING 
SELF APPEARS TO BE LOCATED OUTSIDE THE BODY. 
THE PERSON NEVER REFERS TO ANY DOPPELGÄNGER, 
BUT REPORTS SEEING THE OWN BODY FROM OUTSIDE 


(LEFT PANEL FROM THE COVER OF BLACKMORE, 1982). 

THIS ILLUSORY EXPERIENCE CAN BE ELICITED BY ELEC 
TROCORTICAL STIMULATION, ESPECIALLY OVER THE 
RIGHT TEMPORO-PARIETAL JUNCTION, A MULTISEN- 
SORY AREA FREQUENTLY AFFECTED IN NEUROLOGI- 
CAL PATIENTS, WHO REPORT OBES. HOWEVER, OBES 
CAN ALSO BE EXPERIENCED BY HEALTHY PERSONS, 
ESPECIALLY IN LIFE-THREATENING SITUATIONS. THEY 
CAN BE SIMULATED BY WATCHING A MOVIE OF ONE- 
SELF TAKEN FROM BEHIND AND FED ONLINE INTO A 
HEAD-MOUNTED DISPLAY. THE SIMULATION IS MOST 
EFFECTIVE IN REAL LIFE SITUATION 


(TOP RIGHT PANEL: THE AUTHOR WALKING THROUGH THE STREETS OF COLOGNE; 
FROM THE MOVIE “DER ZWEITE LEIB” BY DANIEL MULLER, KOLN, 2003), 


BUT CRITICAL FACTORS CAN BEST BE CONTROLLED 
IN A LABORATORY SETTING 


(BOTTOM RIGHT PANEL ADOPTED FROM LENGGENHAGER ET AL., 2007). 


facing death. And it is the universal occurrence of OBEs under these circumstances (Sheils, 
1978) that may console whole subcultures in contemplating the mortality of the person after 
bodily death. 


NEUROLOGICAL AND DIGITAL 
DOPPELGANGER: POINTS OF 
CONTACT 


Avatars are doppelganger in the sense that they are digital representations of a person’s 
self. Rather than brain-created, they are created by a computer algorithm, which simulates 
the presence of a second self. This simulation comprises different components. In part, the 
feeling of relatedness toward one’s avatar rests on visual similarity. By uploading self-face 
photographs, players can make a generic character take their own appearance. Such photo- 
realistic resemblance not only increases identification with the avatar but also leads to an 
enhanced Proteus effect (BOX; Yee and Bailenson, 2007). This effect refers to a change in a 
player’s behavior caused by the type of avatar used; identification with an attractive avatar, 
for instance, makes a player more gregarious and behave more socially. Age-progressed 
avatars make players more future-oriented and increase saving behavior (Hershfield et al., 
2019). However, visual similarity based on facial configurations, is just one thing. Its contri- 
bution to the feeling of embodiment is relatively modest. As in an autoscopic hallucination, 
a doppelgänger-avatar merely mirroring one’s facial identity will not be embodied persua- 
sively. The sense of embodiment, a key factor for both the neuroscience of the self and VR 
game design, relies on three dimensions (Blanke and Metzinger, 2009); self-location and 
corresponding perspective (e.g., if I feel located in a body and perceive the environment 
from that particular point in space), ownership (I experience the body and its single parts 
as my own), and agency (I can move the body intentionally). VR simulations of OBEs 
quantify the degree of embodiment by “proprioceptive drift”, that is the indicated shift of 
self-location away from that of the real body towards the virtual body. Such drift is induced 
by having research participants observe themselves being touched on the back in a real-time 
clip taken from behind (Lenggenhager et al., 2007; cf. Fig. 4). Although visual similarity 
does play a role (there is no proprioceptive drift towards a non-anthropomorphic object), 
the key factor is visual-tactile synchrony and synchrony of the intention to move an avatar’s 
limb and instantaneously seeing it move accordingly. Synchrony-dependent multisensory 
and sensory-motor integration is necessary for an optimal identification with any object, 
being it one’s real body, a doppelganger or an avatar. Accordingly, designers of immersive 
VR games strive to add more senses than just vision to the experience of exploring digital 
environments—from haptics to 3D audio and even to scent and taste (Ranasinghe et al., 
2017). In the future, adjustment of an avatar’s heartbeat such that it follows its owner’s 
rhythm may further contribute to the emergence and sustainability of embodiment (Aspell 
et al., 2013). 

What the study of autoscopic phenomena unequivocally shows is that the observer’s 
perspective matters a great deal. While a disembodied perspective experienced during 
fright-induced OBEs signals the loss of concern about the body’s fate, perspective-taking 
also matters in dreams and personal memories. Dangerous scenes or fear-loaded memo- 
ries are typically remembered from a disembodied, third-person perspective while neu- 
tral or positive-emotional memories are remembered as originally experienced, i.e. from a 
within-body location (Nigro and Neisser, 1983). There is experimental evidence that cues 
encoded during a simulated out-of-body perspective are less well remembered than the same 
cues picked up from a body-centered first-person perspective (Bergouignan et al., 2014). 
These are neat illustrations of embodied cognition. ‘They also give testimony to the inter- 


actions between emotion and space. Clinical wisdom has it that, statistically, benevolent 
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This rule, quantitatively supported 
in the case of visual hallucinations 
(Walters et al., 2006) is one mani- 
festation of a more universal law, 
valid in all humans and animals 
with a bilateral body symmetry 
and right-sided motor preferences: 
the association of “good” with 
“right” and of “bad” with “left” 
(Tamagni et al., 2009 for the 
literature). 


doppelgänger approach a person primarily from the right side of space while malevolent 
reduplications rather prefer to manifest themselves along the left side of one’s body”. Game 
design could profit for implementing algorithms, which allow a sudden switch of perspec- 
tive depending on situational details in a game. 

As emphasized by Bailenson (2012), what makes a conventional avatar a doppel- 
gänger avatar is the ability to act independent of the associated human agent. The psy- 
chological effects of such “avatars who ignore their owners” (Bailenson and Segovia, 
2010), are manifold and powerful. Fortunately, the negative, self-destructive properties of 
brain-derived doppelgänger reviewed above will not pay out for the gaming industry nor 
for other commercial VR applications. There are many ways to profit from a potentially 
beneficial impact of digital doppelgänger (Rheu et al., 2020, for a review of the use of 
avatars to promote health behavior). Imagine you immerse in a game-like video observing 
your personalized avatar in everyday situations. It is programmed to gain weight after a 
week you neglected physical exercise and to loose weight once you vigorously trained. There 
is empirical evidence that your exercise behavior can be positively influenced after receiv- 
ing such vicarious reinforcement by your doppelganger (Fox and Bailenson, 2009). “Avatar 
therapy” (Gerner, 2020, for a philosophically informed overview) provides a growing means 
to treat a broad range of psychological disorders, from phobias to hallucinations and other 
psychotic symptoms. Perhaps, it will also prove useful to treat “digital depersonalization” 
(Bezzubowa, 2020), a confusion between real and virtual self, which may manifest after 


repeated shifts between physical and virtual realities? 


SOME TECHNICAL TERMS FROM 
CLINICAL AND COMPUTER SCIENCE 


Autoscopic Phenomena A class of illusory reduplications of body and self. 
Comprises the four types of doppelganger discussed 
in the present chapter plus negative heautoscopy 
(not seeing oneself in a mirror) and inner heautoscopy 
(visualization of inner organs in extracorporeal space). 


Bodily self The experience of oneself as an embodied being, who 
perceives the environment from a particular location 
in space with a corresponding perspective and whose 


motions follow one’s intentions. 


Coenesthesia The general awareness of one’s body arising from 
aggregated non-visual, mostly interoceptive impressions 


about one’s bodily state. 


Cognetics A branch of robotics that unites robotic technology 
and the cognitive neuroscience of bodily awareness 
and self-consciousness to develop machines capable to 
accommodate self-other differentiation, empathy and 


social interactions. 


Proteus effect The modulation of a person’s behavior by properties 


(perceptual or personality-related) of her avatar. 


Transitivism The projection of a patient’s symptoms onto other 


persons or a doppelganger of herself 
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THE SECRET BEHIND 
THE ACTOR-AND- 
AVATAR FMRI-STUDY 


LORENA KEGEL 


Computer-generated faces, or avatars, are often used for commercial purposes in advertising 
and service industries or scientific applications. Using advanced software, it is possible to 
create human-like avatars that look and move highly realistic (de Borst/de Gelder 2015). 
However, we do not know if we perceive human-like avatars in the same manner as other 
human beings. How does the artificialness of avatars influence brain processes that underlie 
our behavior towards them? With our research, we work towards closing this knowledge gap 
and study the difference between the processing of facial expressions shown by human-like 
avatars or actors. This is realized using a technology called functional magnetic resonance 


imaging, which is described in the next section. 


WHAT IS FUNCTIONAL MAG- 
NETIC RESONANCE IMAGING 
AND WHY ARE WE USINGIT? 


Functional magnetic resonance imaging (fMRI) is a variant of magnetic resonance imaging 
(MRI). Both are imaging techniques that allow us to study the brain from the outside while 
it is in action. When we are using fMRI, we create images of how the brain is working, 
what we refer to as function. When we are using MRI, we create images of what the brain is 
made of, what we refer to as structure. In combination, both techniques enable us to under- 
stand how the brain is working while we perform a cognitive task, experience an emotion, 
or simply rest and let our thoughts wander (Poldrack 2018). This interest in understanding 
how the brain works has been the motivation for many fMRI studies, as it is certainly one 


of the most challenging scientific enigmas to date. 


WHAT DO WE MEASURE 
WITH IT? 


Measurements with {MRI exploit the fact that nerve cells in our brain need to be constantly 
supplied with energy. This energy reaches the nerve cells through the blood vessels in the 
form of oxygen and sugar. When we perform a cognitive task or experience an emotion, 


certain regions in the brain become active and the nerve cells in these regions consume more 


energy than in a state of no activation. This increased energy demand is in turn associated 
with an increased blood flow, which is measurable with fMRI via the magnetic properties 
of hemoglobin in the blood (Poldrack/Mumford/Nichols 2011). 


HOW IS IT PERFORMED? 


In order to create images of how the brain works using fMRI, we need to perform several 
measurements. In the beginning, we start with a solely structural measurement that maps 
the shape and structure of the brain. The structural images are needed so that the brain 
activation, which is measured during the functional measurements, can be overlaid on them 
to show its location in the brain (Soares et al. 2016). 

After the structural measurement, functional measurements are conducted to obtain 
images of brain activation. Usually, brain activation is measured in two different states: in 
a state that we want to investigate and, in a state that we want to compare with the state of 
interest (Poldrack 2018). In our study, we are interested in how the brain is working while 
people see an avatar’s face compared with brain activation while people see an actor’s face. 
When contrasting both conditions, we can see which brain regions are active more strongly 


. ’ > 
in response to an avatar s face or an actors face. 


WHAT DO WE EXPECT FROM 
OUR FMRI MEASUREMENTS? 


In order to understand how the brain is working while people see an avatar or an actor’s face, 
it is important to go back to previous studies investigating brain activation in response to 
human faces per se. Such studies have revealed that face processing relies on a bilateral net- 
work of brain regions in the temporal and frontal lobe of the brain (Duchaine/Yovel 2015, 
Haxby/Hoffman/Gobbini 2000): the fusiform face area in the fusiform gyrus, the superior 
temporal sulcus, and the inferior frontal gyrus. In our study, we can also show that these 
regions consistently become active when participants are looking at the faces of actors. A 


typical activation pattern is outlined in Figure 1 (showing in color and encircled). 
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FIG.1 


ILLUSTRATION OF THE TYPICAL BRAIN RESPONSE 
TO HUMAN FACES. EACH IMAGE SHOWS THE BRAIN 
ACTIVATION OF ONE PARTICIPANT OF THE FMRI STUDY. 


55 


56 


Previous studies have also shown that the amygdala plays an important role while people 
look at faces, especially when the faces show an emotional expression. The amygdala is 
essential to people’s ability to feel certain emotions and to perceive them in other individuals 
(Adolphs 2008). Hence, it is bilaterally active in response to faces showing an emotional 
expression. Such a bilateral activation pattern in the amygdala of one of our participants is 
outlined in Figure 2. 

If the amygdala or other regions in the temporal lobe are altered due to a neuro- 
logical disease, the entire brain network that is active when people see a face may be affected. 
Previous studies have shown that this is the case in 
people with epilepsy that originates in the temporal 
lobe of the brain (Ives-Deliperi/Jokeit 2019). For 
example, it was found that the amygdala is less active 
in response to a human face in the brain hemisphere 
where the epileptic seizures originate compared with 
the amygdala in the not affected brain hemisphere 
(Ives-Deliperi/Butler/Jokeit 2017, Labudda/Mertens/ 
Steinkroeger/Bien/Woermann 2014, Schacher et al. 
2006; Toller et al., 2015). Therefore, a further aim of 
the {MRI study is to compare which brain regions 
are active while people with temporal lobe epilepsy 


(TLE) and people without epilepsy look at actor and 


avatar faces. 
Amygdala 
FIG. 2 


ILLUSTRATION OF THE TYPICAL AMYGDALA RESPONSE TO HUMAN 
FACES WITH A FEARFUL EXPRESSION. THE IMAGE SHOWS THE 
BRAIN ACTIVATION OF ONE PARTICIPANT OF THE FMRI STUDY. 


METHODS OF 
THE FMRI STUDY 


For the study, we used a set of videos that have been developed in a four-step process in 
cooperation with the Zurich University of the Arts: 
1. Fearful and neutral facial expressions were recorded from female and male actors. 
2. For two female and to male actors, a customized avatar was created by a graphic artist 
to match their appearance. 
3. By motion tracking, the actors’ recorded expressions were conveyed onto their avatar 
faces. 
4. The recorded material was divided into single videos of three seconds duration and 


the best 128 videos were selected to show during the fMRI measurement. 


26 participants without epilepsy and 17 partcipants with epilepsy took part in the fMRI 
measurements. They were scanned with an MRI scanner of the Medizinisch Radiologisches 
Institut located at the Schulthess Clinic in Zurich. When lying in the MRI scanner, par- 
ticipants watched the videos of actor and avatar faces via a back-projection that was visible 


by a mirror above their eyes (see Figure 3; the table with the participant is positioned outside 


the tube of the MRI scanner for illustration purposes). While images of the brain were 
created, participants were required to lie as motionless as possible inside the MRI scanner. 
The measurement took around 35 minutes. Before each video, a fixation cross was pre- 
sented, so participants knew where the video would appear. After each video, a black screen 


was presented until the next fixation cross appeared announcing the subsequent video (see 


Figure 4 for illustration). 


Fixation cross Black screen Video 
600 — 800ms 3000-5000ms 3000ms 


| 


Video Fixation cross Black screen 
3000ms 600 - 800ms 3000-5000ms 


FIG. 3 FIG. 4 
THE FMRI SETTING AT THE EPILEPSY ILLUSTRATION OF THE 


CENTER AT KLINIK LENGG, ZURICH. PROCEDURE OF THE FMRI STUDY 


RESULTS OF THE FMRI STUDY 


Before the results are outlined in detail, we want to summarize our three main findings. The 

first two findings are based on the results found in participants without TLE and the third 
finding is based on the comparison between participants with and without TLE: 

1. While participants look at avatar faces with a neutral expression, the brain is active in 

a similar manner than when people look at faces of actors with a neutral expression. 

2. While participants look at avatar faces with a fearful expression, certain brain regions 

are less active than when participants look at faces of actor with a fearful expression. 

3. The difference in response to fearful avatar faces and fearful actor faces is smaller in 


participants with TLE than in participants without TLE. 


Concerning the second finding, an activation difference between fearful expressions of 
avatars and actors emerges in the superior temporal sulcus and the inferior frontal gyrus. 
Both regions are sensitive to human faces and human facial motion, which also has been 
shown in previous studies (see again Figure 1; Duchaine/Yovel 2015, Haxby et al. 2000). In 
other words, avatar faces do not seem to activate these brain regions in the same way as their 
human counterparts. However, as the first finding shows, this difference between avatar 
faces and faces of actors is only present if the faces show an emotional expression. 

When comparing the brain activation in participants with TLE to participants with- 
out TLE, we see that several regions that are associated with the processing of faces are less 
activated in participants with epilepsy. For example, the amygdala (see again Figure 2) of 
participants with TLE shows a smaller response to avatar faces and faces of actors with a 


fearful expression than the amygdala in participants without TLE. 
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SUMMARY AND OUTLOOK 


In summary, some regions in our brain work differently when we look at actor faces than 
when we look at avatar faces. Furthermore, if epilepsy is present, it influences these brain 
activation patterns. It is likely, that these differences in response to avatar faces also have 
an influence on our behavior during interactions with avatars. This has important conse- 
quences for the use of avatars in various commercial and public applications and research. 
Interestingly, avatars and computer-generated characters in general are increasingly being 
used in diverse fields. As a result, with every day we become more experienced with such 
characters. How will this increasing exposure with computer-generated characters influence 
our behavior towards them? How will our brain adapt to this? At this point, we note that 
we currently have no answers to these questions and are in need of future studies to address 
this. 
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AVATAR’S 
FACES IN EEG 


TERESA SOLLFRANK 


In our daily lives emotions play an important role in how we think and behave. The emotions 
we feel can compel us to take action and the recognition of emotions enables us to interact 
with others and adapt our behaviour to the circumstances. Particularly facial expressions 
contain a lot of information that allow us to interpret emotions and analysing the brain pro- 
cesses during recognition of different facial emotions (e.g. anger, happiness, fear) can help 
us to better understand emotion recognition and social cognition. 

Sometimes the quality in emotion recognition can be diminished due to neurological 
diseases like epilepsy. In our study, we plan to compare the reaction towards fearful and 
neutral faces in actors and computer generated faces, so called avatars between healthy sub- 
jects and people who suffer from epilepsy. Our aim is to better understand the processing of 
emotion recognition in faces in the neural system and the effects of avatars on the quality of 


emotion perception. 


EPILEPSY 


Epilepsy is defined as a condition of recurrent unprovoked seizures that start in the brain. 
These seizures are brief lapses of attention or brief episodes of involuntary movement or sen- 
sations that may involve a part of the body, the entire body and are sometimes accompanied 
by loss of consciousness. Seizures are a result of excessive electrical discharges in a group of 
brain cells and the characteristic of a seizure depends on where in the brain the disturbance 
starts, and how far it spreads. The causes can be complex and sometimes hard to identify, but 
may include a genetic tendency or a structural change in the brain (e.g. a stroke, head injury 
or tumor). Around 50 million people worldwide have epilepsy, making it one of the most 
common neurological diseases globally. Usually medication with anti-epileptic drugs is the 
most important and common treatment, an up to 70% of the people can become seizure-free 
with medication. Other treatment options are for example brain surgery, deep-brain stimu- 
lation, vagus-nerve stimulation or a ketogenic diet. 

Deficits in emotion recognition and social cognition are especially reported in people 
suffering from temporal lobe epilepsy (TLE) (Meletti et al. 2003, 2009, Bonora et al. 2011, 
Broicher et al. 2012, Amlerova et al. 2014, Steiger/Jokeit 2017). TLE is a chronic disorder 
characterized by recurrent focal seizures that originate in the temporal lobe of the brain. The 
primary functions of the temporal lobe are to process sensory information and derive it into 
memories, language, and emotions. Located within the temporal lobe are the hippocampus 
and the amygdala, which are part of the limbic system. ‘The hippocampus manages the for- 


mation of new memories and communicates closely with the amygdala, which is responsible 


60 


for the processing of emotions. Parts of the temporal lobe (fusiform gyrus, parahippocampal 
gyrus) are needed for the processing of visual stimuli to allow us to recognize objects or 
faces. Our temporal lobes are essential to process emotional information and impairments 


in these structures seem to be connected to deficits in emotional perception. 


ACTOR VS AVATAR 


Investigating emotional perception in an experimental environment demands meaningful 
and realistic stimuli material. The quality of emotional information depends on the pictures 
and videos presented that contain facial expressions. Recent studies have already shown that 
it may be helpful to use avatars to examine and possibly enhance emotion recognition in a 
variety of conditions like autism (Bekele et al. 2014, Hopkins et al. 2011) and schizophrenia 
(Dyck et al. 2010). However, while emotional faces of avatars have been shown to elicit 
amygdala activation comparable to that elicited by human faces in healthy participants, 
responses to human faces in face-sensitive cortical structures were found to be significantly 


stronger (Moser et al. 2007). 


IDEA BEHIND OUR PROJECT 


The overall purpose of the EEG study was to learn about the basis of processing of dynamic 
emotional expressions in human as well in avatars. 

In order to achieve this, we compared brain electrical responses elicited by fearful 
and neutral facial expressions of actresses/actors and their avatars. The evaluation of the 
difficulties associated with processing of facial expressions makes it absolutely necessary to 
use valid stimuli that fully capture the facial and emotion related information displayed in 
a face. However, most studies on the perception and recognition of facial expressions have 
used static pictures displaying different emotional expressions. Only few studies applied 
dynamic facial expressions, which are considered to be more ecologic and therefore closer to 
daily life (Sarkheil et al. 2013). Yet designing and generating dynamic facial stimuli poses 
the problem of controlling for temporal und figural properties of the face and the developing 
facial expression. A promising way to bypass this is the use of computer-generated avatars, 
which allow to form and systematically control important features of the facial expression. 
We hypothesise that there will be a significant differences in both people with epilepsy and 
healthy controls elicited by facial expressions of actresses/actors and their avatars. 

The secondary objective was to differentiate between brain electrical responses of 
people with and without epilepsy to fearful and neutral facial expressions. In particular, 
we invited patients diagnosed with temporal lobe epilepsy to take part in our study. We 
hypothesise that the responses to fearful facial expressions are significantly reduced in 
people diagnosed with temporal lobe epilepsy as compared to healthy controls, as it has 
been shown in earlier functional MRI studies (Schacher et al. 2006, Broicher et al. 2010). 


EEG: WHAT CAN YOU 


MEASURE? 


At the root of all our thoughts, emotions and behaviours is the communication between 


neurons within our brains. The billions of nerve cells in your brain produce very small 


electrical signals that form patterns called brain waves while they communicate with each 


other. An ‘electroencephalogram’ (EEG) is a test used to evaluate the electrical activit 
phalog y 


in the brain and is considered by many to be one of the most efficient and relatively inex- 


pensive methods for examining activity in the brain. It provides excellent time resolution, 


allowing us to detect activity within cortical areas, even at sub-second timescales. It tracks 


and records these brain wave patterns of varying frequency and amplitude, measured in 


voltage. These EEG waveforms are generally classified according to their frequency, ampli- 


tude, and shape, as well as the sites on the scalp at which they are recorded. 


Our brainwaves change according to what we’re doing and feeling. When slower 


brainwaves are dominant we can feel tired or dreamy. The higher frequencies are dominant 


when we are alert and awake. Altered brain waves can be measured in several types of brain 


disorders. For example, when epilepsy is present, seizure activity can appear as rapid spiking 


waves on the EEG. The EEG test can also be used to diagnose other disorders that influence 


brain activity, such as Alzheimer's disease or sleep disorders. Electrical activity in the brain 


can also be measured as a response to an external stimulation of e.g. sight, sound, or touch 


in an experimental condition. These are called evoked potential studies. 


HOW IS IT USED? 


A varying number of small flat metal discs called electrodes are attached to the scalp with a 


conductive gel, paste or dry. The majority of signals captured by EEG represent the summa- 


tion of cortical pyramidal cells in the upper layer of the brain. The electrodes on the scalp are 


very sensitive and they detect these dipoles formed by ten to fifty thousand neurons. EEG 


poorly measures neural activity that occurs below the upper layers of the brain (the cortex). 


The charges picked up by the electrodes are amplified and send to a computer that records 


the results. 

The electrical impulses in an EEG record- 
ing look like wavy lines with peaks and valleys 
(fig. 1). Analysing these data can get quite chal- 
lenging. Signal processing, artefact detection and 
attenuation, feature extraction, and computation 
of mental metrics such as workload, engagement, 
drowsiness, or alertness all require a certain level 
of expertise and experience to properly identify 
and extract valuable information from the col- 


lected data. 
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EXAMPLE OF EEG SIGNALS. EACH LINE REPRESENTS 
THE SIGNAL MEASURED AT A SPECIFIC ELECTRODE 
LOCATION ON THE SCALP. 


61 


m 


FIG. 2 
EXPERIMENTAL SET-UP 
AT THE EPILEPSY 
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THE STUDY 


We invited 10 healthy controls and 10 TLE subjects to participate in our study. Subjects sat 
in front of a screen in a comfortable position, watching videos and pictures of the actresses/ 
actors and avatar stimuli material while continuous EEG signals were recorded with twenty- 
one sintered Ag/AgCl scalp electrodes that were placed on the head of each subject (fig. 2). 


FIG. 3 


CENTER AT KLINIK 


LENGG, ZURICH. PROTOCOL 


In the first part of the experiment 
subjects watched 120 brief videos 


(2-3 sec) of faces of actresses/actors 
and their avatars showing neutral as 
well as fearful faces on a computer 


screen in randomized order, sepa- 


STIMULI ITERATION. EACH SINGLE 
ITERATION OF A FIXATION CROSS 
FOLLOWED BY THE VIDEO/PICTURE 
FOLLOWED BY A BLANK SCREEN IS 
LABELLED ASA TRIAL. 


rated from each other respectively by 4sec of blank screen with a fixation cross in the middle 


of the screen (fig. 3). 


Afterwards subjects did a second round with the same sequence but videos were 


replaced with pictures that have been taken from each clip. To keep subjects engaged, they 


were asked to perform an attentional task by pressing a button during a specific control 


condition. 


RESULTS 


For analysing the EEG datasets, the trials for each condition (ACTOR neutral, ACTOR 
fear, AVATAR neutral, AVATAR fear) are averaged and the signal is processed. 


Is there a difference in the brain electrical response between watching 
actresses/actors and their avatars? 
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EEG signals were measured while subjects watched subsequent pictures of actors and their 
avatars with neutral and fearful facial expressions. ‘The trials of each condition (i.e. actor or 
avatar with neutral/fearful faces) were then averaged to measure the specific brain response 
to the different faces, this response is called an event-related potential (ERP). 

As an example, the measured EPRs to the four different conditions of 5 healthy 
volunteers are shown in fig 4. The waveform represents the time locked reaction of the brain 
(measured over CZ) within 1000 ms after the pic- 
tures are presented. The blue waves show the reaction 


to faces of actors with neutral (light blue) and fearful 


(dark blue) expression, the red curves the reaction to 


faces of avatars with neutral (light red) and fearful 


— Actor Neutral 
= Actor Angst 

== Avatar Neutral 
—— Avatar Angst 


(red) expression. å 


We can see a fast reaction to the different 


stimuli within only 100-200 ms after presentation 
and the waveforms seem to differ after 200 ms depen- |£ 
dent on whether the picture of an actor or avatar is | y 
presented. A more long-lasting change between the 


different curves is seen after 300 ms. 
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FIG. 4 mee 
EPRS TO THE FOUR DIFFERENT CONDITIONS 
ANALYSED FOR 5 HEALTHY VOLUNTEERS. 


Is there a difference between brain electrical responses of people with and 
without epilepsy to fearful and neutral facial expressions? 


EEG signals were measured while subjects watched the 120 video clips and subsequently 
the trials of each condition were averaged. Afterwards a so-called time-frequency spectrum 
(fig. 5) was computed, where we calculated the intensity of power (dB) in each frequency 
over the period of time while the subjects watched the videos. As an example, we can have a 
closer look on the condition ACTOR fear. The more 


activated the brain, the more power is produced and 
the more intense the colours of the heat map become. ACTOR fear 


The intensity of activation is more pronounced in — TE 
Power (30) 
25 
0 
-2.5 


healthy subjects compared to the epilepsy subjects in 


our study. 
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FIG. 5 

TIME-FREQUENCY SPEC- 
TRUM OF POWER AT A 
CENTRAL ELECTRODE. 


HC= HEALTHY CONTROL GROUP, TLE = SUBJECTS WITH TEMPORAL LOBE 
EPILEPSY. THIS IMAGES WERE CALCULATED FOR ELECTRODE POSITION C4. 
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neutral 


Mean power (dB) 


FIG. 6 S 


In a second step, we are able to transform 
the values of these heat maps into a numeric 
value during a specific time window (500- 
2000ms) and frequency range (8-13 Hz) 


for all conditions. These values of power 


neutral 


reflect the intensity of electrical response 
and the results are depicted in fig. 6. The 
responses to fearful facial expressions seem 
to be reduced in people diagnosed with 
temporal lobe epilepsy as compared to 


healthy controls. 


POWER VALUES OF 
ALL CONDITIONS 
FOR HEALTHY AND 
TEMPORAL LOBE 
EPILEPSY SUBJECTS. 


THE VALUES WERE MEASURED 
FOR A TIME WINDOW OF 500- 
2000 MS IN THE FREQUENCY 
RANGE FROM 8-13 HZ. 


CONCLUSION 


So far the EEG measurements suggest that the faces of actresses/actors are perceived 
different than faces of avatars not dependent on a fearful expression. Fear in faces of actors 
seem to affect us different than in faces of avatars although further analysis is needed. 

In previous studies a reduction in brain electrical activity during emotional percep- 
tion could be seen in epilepsy patients (TLE). Our preliminary results are reinforcing these 
findings for fearful facial expressions. In total, we are aiming for a group size of 26 subjects 
to have statistically meaningful results. Measurements are ongoing and results will be shared 


with the academic community by publishing an article in a topic related scientific journal. 
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“DOPPELTGANGER”. 
BODY DOUBLES, 
OR THE SHADOW 
AND HIS EGO 


MICHAEL MAYER 


“Oh what is that grazing the banister? 

Is that not how I looked out the looking glass? 
And those are my limbs—how dazzling! 

Now it lifts up its hands, like tufts of yarn. 
That’s the line of my brow and my curls,— 
Woe, am I mad, or is my end drawing near?” 


ANNETTE VON DROSTE-HULSHOFF, 
DAS FRAULEIN VON RODENSCHILD, 1842 


FIG. 1 

FRIEDRICH HUNDT, 
PORTRAIT OF ANNETTE 
VON DROSTE-HÜLSHOFF 


DAGUERREOTYPE, 1845 


“It was a little before eight o'clock in the morning when Yakov Petrovitch Golyad- 

kin, a titular councillor, woke up from a long sleep.” (Dostoyevsky 1951: 477) 

That is the opening of the novel The Double, which the young Dostoevski pub- 
lished after the runaway success of his debut, Poor Folk (1846). The novel’s beginning is 
unspectacular—unlike the events to follow. The book’s action spans four days of a rainy, 
cold November in mid-nineteenth-century St. Petersburg. To summarize: at a party, on a 
renewed attempt to win the favor of Klara Olsufyevna Berendeyev— a young woman from 
an élite family whom he venerates— Golyadkin is publicly humiliated and expelled from 
the house. In deep despair, vexed by self-recrimination and suicidal thoughts, he wanders 
around nighttime St. Petersburg for hours. Finally, he catches sight of a lone pedestrian 
who strikes him as strange and uncanny. Acting on a spontaneous impulse, he follows the 
stranger all the way to the man’s apartment, which— to Golyadkin’s great surprise — turns 


out to be his own. His sense of foreboding soon bears out: 
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The stranger, also in his coat and hat, was sitting before him on his bed, and with 
a faint smile, screwing up his eyes, nodded to him in a friendly way. Mr. Golyadkin 
wanted to scream, but could not-to protest in some way, but his strength failed him. 
His hair stood on end, and he almost fell down with horror. And, indeed, there was 
good reason. He recognized his nocturnal visitor. The nocturnal visitor was no other 
than himself—Mr. Golyadkin himself, another Mr. Golyadkin, but absolutely the same 
as himself-in fact, what is called a double in every respect. (Dostoyevsky 1951: 515) 


After some initial hesitation, the original Golyadkin invites his alter ego to spend the night. 
Despite their early apprehensions, the two of them pass the evening in a more relaxed, 
almost amiable mood. In the process, it emerges that the double is not only identical to 
Golyadkin in every way, but he even comes from the same area, has the same name, and 
finally —as we learn the next day— works in the very same department of the civil service, 
where he is greeted warmly as a new member of staff. Although conversations with col- 
leagues indicate that they have noticed the unusual duplication of person and name, this is 
not further remarked upon. Within several days, the “real” Golyadkin’s hopes for a mutually 
beneficial relationship with his alter ego fall apart. The “fake” Golyadkin grows increasingly 
rude, scheming, and at last openly malicious. A disastrous rivalry develops between the 
original and the copy, between Golyadkins I and II, a battle that Golyadkin I ultimately 
loses. His doppelganger wins out as a successful second edition; Goldyadkin I had previ- 
ously only dreamed of winning his colleagues’ sympathies and his boss’s recognition. All his 
attempts to defend against being gradually displaced from his own life are in vain. 

Yet that is not all. At another soirée at the Berendeyevs’ parlor and in front of 
the daughter, with whom Golyakin is secretly in love, the situation derails altogether. 
Golyadkin is publicly exposed and sent away once and for all. Several gawkers run alongside 
the carriage that is carting him off. One by one, they fall behind, leaving only a single man, 
“Mr. Golyadkin’s unworthy twin”: 


With his hands in the trouser pockets of his green uniform he ran on with a satisfied 
air, skipping first to one and then to the other side of the carriage, sometimes 
catching hold of the window-frame and hanging on by it, poking his head in at the 
window, and throwing farewell kisses to Mr. Golyadkin. But he began to get tired, 
he was less and less often to be seen, and at last vanished altogether 

(Dostoyevsky 1951: 614). 


The doctor, who is alongside Golyadkin in the carriage, pronounces—like “a judge’s 
sentence” — that he is being confined to an institution. The novel ends with Golyadkin 


shrieking. 


Dostoevsky’s novel was an unexpected flop with readers, and the critics tore it 

apart. Yet he maintained mixed feelings about it. Throughout his life, he viewed 

it as a failure, but still he wrote that “its idea was rather lucid, and I have never 
expressed in my writings anything more serious.” (Dostoyevsky 1919: 883) Aside from the 
complex form and portrayal, the novel has some unusual features that, to my knowledge, 
have never been examined in the context of the doppelgänger motif. This doppelganger 
does not simply disappear at the end. He is left behind. By the time the original Golyadkin 
is sent away, his clone has fully taken over his life—his personal, professional, and social 
position —and takes his place as an improvement on this original. Golyadkin II has utterly 
absorbed Golyadkin I. If the doppelgänger— or indeed Doppeltgänger (doubled-goer), as the 
German word is spelled in the Grimm Brothers’ famous dictionary—is a person who can 
“appear in two different places at once”— Dostoevsky resolves the conflict between them 


not only by segregating the two editions of the same person in separate locations, but by 


making one of them literally vanish from view: Golyadkin I, as an institutionalized patient 
(Grimm 1860: 1263). Golyadkin II is left behind as an ambitious bureaucrat. The struggle 
is over, and the roles of winner and loser have been clearly assigned. 

The agonal confrontation between a self/ego and its doppelganger is already visibly 
deviating here from its antecedent in German Romanticism. Specifically, E.T.H. Hoffmann 
invoked “internal struggles” waged by “some dread power which endeavors to destroy us in 
our own selves” against those very selves, so as to externalize that internal struggle and to 
project it into the real world as a feud between two symmetrical, mirror-image combatants 
(Hoffmann 2008: 14). Thus, the doppelgänger figure appears to presage the “nuclear fission 
of the ego” that the individual (Greek dtamos) — following the literal meaning of “indivisi- 
ble” — would undergo in the twentieth century and from which it would never fully recover. 

Indeed, beginning with the turn of the nineteenth century, these sorts of doubling 
fantasies seemed to run rampant: besides in the work of the authors already mentioned, they 
could be found in the pages of Jean Paul, Heinrich Heine, Theodor Storm, Edgar Allen Poe, 
Oscar Wilde, and others. The identical replication of oneself becomes an occasion to inves- 
tigate what is presumably the most familiar, the se/f as the stage of radical se/f-estrangement 
(Freud 2001: 3696). Of course, Sigmund Freud saw in this the workings of the unconscious 
that governs our minds. Specifically, he considered the “recurrence of the same thing” to 
be “what achieves such an indubitably uncanny effect,” as expressed most prominently in 
the doppelganger motif (Freud 2001: 3696). Drawing on Otto Rank (1993), he stresses 
the fundamental ambivalence toward the doppelgänger, who seems, on the one hand, to 
threaten the ego and the ego identity, but on the other to provide “insurance against the 
destruction of the ego” through duplication (Freud 2001: 3687). We see this ambivalence, 
a classic symptom of any psychiatric condition, at the beginning of Dostoevsky’s novel in 
Golyadkin’s ambivalent relationship with his clone, which, over the course of their first 
encounter, vacillates between a diffuse apprehensiveness and hopes of friendly coexistence. 
Only after the behavior of Golyadkin’s double grows increasingly hostile does the nature of 
their relationship become clear. The duplicate becomes a lethal threat, the “evil self.” Hence, 
the doppelganger motif took on another attribute: a qualitative discrepancy between good 
and evil that would culminate forty years later in Robert Louis Stevenson’s Strange Case 
of Dr. Jekyll and Mr Hyde (1886), which became world-famous in both literature and film. 
However, in Dostoevsky’s novel, the complete absorption of the original by his update, who 
has been strategically and professionally optimized, raises a host of questions that I cannot 
altogether untangle here. I would therefore like to outline at least one of these questions and 


reflect on it analytically in light of our own time. 


Scholars of literary, cultural, and media studies were quick to notice the temporal 

proximity between the emergence of doppelganger themes in Romantic Era prose 

and poetry and the invention of photography in the early nineteenth century 
(Starl 2011). As early as 1844, the Westphalian poet Annette von Droste-Hülshoff associated 
a ghostly encounter with a doppelgänger with “Daguerre’s pictures” in her poem “Doppelt- 
ganger.” (Droste-Hülshoff 2014: 384) People were once shocked by the optical-chemical 
production of their own likenesses in daguerreotypes and, later, the limitless reproducibility 
of such likenesses through photography. The very capability of freezing any moment in time 
and consigning it to some pictorial medium provoked a rupture in the epistemic regime of 
an era that was beginning to blur the lines between visibility, recognizability, and reality. 
Meanwhile, the capability of creating a photographic likeness of myself transferred this 
rupture to the inner workings of human psychology. Ever since then, we have been con- 
fronted by something at once foreign and familiar, someone who is both me and not me. 
Golyadkin’s cry— “Or pretend that I am not myself, but somebody else strikingly like 
me ... Simply not I, not I—and that’s all” (Dostoyevsky 1951: 481)— could just as easily 


be a reaction to seeing his own photographic likeness. As late as the waning twentieth 
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For more on this, see 
Meyer-Drawe (2018), 26f. 


century, in his final and perhaps most beautiful book, Camera Lucida, Roland Barthes was 
still invoking the “profound madness of Photography,” pointing to “the advent of myself as 
other: a cunning dissociation of consciousness from identity.” (Barthes 1981: 12-13) This is 
a reaction we have by now learned to repress. But to this day, even when looking at trivial 
snapshots, we still feel a remnant of that same animal horror that runs through Golyadkin’s 
bones when he encounters his double. Barthes refers to “that faint uneasiness which seizes 
me when I look at ‘myself’ on a piece of paper,” the same reaction that Siegfried Kracauer 


described as a “shudder” some fifty years earlier: 


A shudder runs through the beholder/viewer of old photographs. For they do not 
make visible the knowledge of the original but rather the spatial configuration of a 
moment; it is not the person who appears in his or her photograph, but the sum of 
what can be deducted from him or her. It annihilates the person by portraying him 
or her, and were person and portrayal to converge, the person would cease to exist. 
(Kracauer 1993: 431) 


The culture of photography has never quite shaken off that uneasiness. The equivocal sensa- 
tion that can always accompany a glance at one’s own photographic likeness points to that 
same dualistic/duellistic confrontation that ultimately ruins the titular councilor Goldyakin, 
and which Kracauer takes to be a literal and overt threat of annihilation. The cryptic 
relationship between photography and death—around which Roland Barthes relentlessly 
circled without ever quite pinning it down— eventually escalates, in Camera Lucida, to the 
question of the dead man who confronts us through his picture as someone living/dead 
(Barthes 2002, 87). Barthes uses the term “punctum” of time to designate that enigmatic 
simultaneity that pervades the photo of the living/dead man: “He is dead and he is going to 
die ... “ (Barthes 1981: 95-96) This was Barthes’s well-known caption beneath Alexander 
Gardner’s photograph of Lewis Paynes, which was taken in 1865 shortly before Payne’s exe- 
cution for his attempted assassination of the US Secretary of State. These words are certainly 
paradigmatic, but they point to the testimonial nature of a recording medium (which may 
well apply to all recording media) that shows the viewer a person who is going to be dead. 
And yet—and here we move slightly beyond Barthes’s horizons —a photographic 
depiction of myself confronts me not only with death and mortality in general, specifically 
someone else’s, but most of all with my own. What faces me in the photographic depiction 
of myself is me—.as the dead man I am going to be. The “crisis of death” —which, in Barthes’s 
view, began “in the second half of the nineteenth century” and must have been connected 
to photograph — perhaps takes on a different contour if we juxtapose the “punctum” of time 
with that singular gaze that I direct at my own likeness (Barthes 1981: 92). What it reflects 
is not in fact another self distinct from me, but myself as the dead man I will someday be, as 


seen through the eyes of someone else I will never be: I am dead and I am going to die. 


Who was it who said that the nineteenth century dreamed up the monsters that 

came true in the twentieth? Apparently, the twenty-first century is taking this a 

step further. A concluding remark on that point. It seems to me that in this age, 
when my self is digitally producible as a virtual avatar in infinitely emulatable environments, 
the doppelganger motif is becoming virulent within humans’ lived experience and realities 
in a new and unsettling form. After all, especially since the worldwide spread of the smart- 
phone in the 2010s, we are now constantly synchronized with digital copies of ourselves. In 
real time, our behavior is compared with and fed back into behavioral patterns extrapolated 
from it. And moreover, via this copy, we are also confronted with digitally optimized ver- 
sions of ourselves, compared to which we are chronically failing short. The digital version sets 
the standards. And, slowly but surely, it takes the place of its analog original, from which it 


was abstracted, so as to reflect an inferiority that confronts it vis-a-vis its digital version.' 


If mathematical game theory took the economic concept of competition as the “enmity 
between individuals” (Engels 1975: 423) who are artificially atomized and pitted against 
one another, then transferred it to a playing board so to formalize it, standardize it algo- 
rithmically, and universalize it as calculated routines of human behavior, these theories’ 
digitalization and automation via Al-based specialized systems have ultimately shifted this 
enmity inside our selves. The social sphere is rendered competitive and hostile following the 
same process applied to the psyche, which now begins to compete with itself within itself 
(see Galison 2001). The market, whose rules were elevated in the mid-twentieth century 
into a guiding principle of knowledge and then to a practical regulating force for the social 
sphere, is mutating into the per-formative force of a subjectivity whose creeping market-ness 
is completing and perfecting the economization of societal, political, social, and finally 
natural life.? And if ethics—at least in the Classical sense, upon which Nietzsche and, in 
his own way, Foucault built— implies that art is “friends with itself,” then a human being 
as self-enemy is quintessentially non-ethical. Dostoevsky’s Golyadkin character seems like 
an earlier prototype of this human being whose se/f-enmity will bear radical implications for 
the question of the subject, its relationship to itself, to the environment, and to the larger 
world. It is as though we no longer trust the ground under our feet and have likewise lost 
faith in the starry sky overhead, our own kind, and ultimately our very selves. It is dizzying 
to contemplate: “At the same time Mr. Golyadkin felt as though the ground were giving way 
under him, as though he were staggering, falling.” (Dostoyevsky 1951: 505) 
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I have started to distinguish this 
process using the neologism 
“homo oeconomicus ludens.” 
See Mayer 2020. 
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THE AVATAR’S 
BODY IN GAME 
SPACES 


STEPHAN GUNZEL 


The phenomenological concept of corporeality is based on the assumption that the body is 
the medium that constitutes space and enables experience: the fact that people have a face, 
which is why they encounter space head-on, and usually stand on two legs, which makes 
them stand upright in space, and that they have a symmetrically organized body that allows 
them to qualitatively differentiate between two sides, defines, according to phenomenology, 
their specific anthropological status. The human body can therefore rightly be referred to 
as the schema or “archetype” of space, in so far as it determines how space appears to the 
individual and thus how spatial perception or the extent to which perception is spatially 
structured. 

The medium of the body can even be spoken of in the full sense of the word, since 
the body medium is in the “middle” in so far as it forms the central starting point for the 
experience of space; but the human body is not to be thought of as a mere mediating instance 
that confronts the inner realm of thought or rationality on the outside, as René Descartes’ 
separation of res extensa and res cogitans suggests. In other words: the mediation performance 
assumed from the phenomenological side has absolute validity and forms the basis of per- 
ception; it is not, for example, deviant from reality because it only mediates certain aspects 
or makes the world accessible in perspective sections. Rather, according to the phenomeno- 
logical view, the perspective is part of reality, the “being-in-the-world”. 

This does not mean, however, that human perception is limited to a way of being- 
in-the-world.— On the contrary, the body schema as the archetype of space can also be 
modified; and this is precisely what technical media, above all image media, make possible: 
they can, for example, extend or limit the range of the body. Rather, the same can be assumed 
for technical media as for the body: even technically mediated structures are not inferior to 
what they represent but show something in a special form. In other words, for media such 
as images in particular—phenomenologically speaking — what they show is of less interest 
than ow they show it. 

From an ontological point of view, however, a representation can have less to do with 
being than with what is represented, since a picture only shows visible qualities, but not the 
tactile or olfactory qualities of things. The “increase in being” (Gottfried Boehm) inherent 
in the picture, if it does, is derived solely from the form of how a content is present: the kind 
of artificial presence can deviate significantly from the way the picture subject appears in its 
actual appearance. Such a phenomenological view of images can already be found among 
the producers of images, especially among painters who understand their activity as an 


exploration of perception. 


According to Maurice Merleau-Ponty, what is special about a still life by the post-impres- 
sionist Paul Cézanne’s, for example, is not that he paints apples on plates with a jug of water 
next to them, but that he paints the plate and the water vessel in a slightly a-perspectival 
manner— thus showing a visual situation, in which one sees the apple on the plate and the 
front of the jug from the point at which the picture places the viewer through the perspec- 
tive of the picture, and at the same time slightly obliquely from above onto the plate and 
into the opening of the jug, both of which are rather round than elliptical. Cézanne has thus 
painted “something”, which in psychology is called “constancy of form”, and this means 
that a person’s perception is not based on a momentary mono-perspective (“photographic”) 
view of things, but on experience of the object: and part of this experience is that plates 
and jugs are usually round rather than elliptical. Cézanne has therefore chosen the mode of 
her life-worldly experience as the form of representation of 
things and it is this mode— the how of representation— in 
which the still life differs from a photograph of an identical 
arrangement. Cézanne is thus a special case of modern art 
because he does not attempt to go beyond the structure of 
life-world experience and produce self-referential or self- 
critical images, as was the case in Cubism, but rather to give 
the experience of space a formal expression in the picture. 
But it is not only painters who can modify the how 
of bodily and spatial perception in the picture or—as in 
the latter case— try to emphasize particular aspects of it by 
modifying it; even the imagination allows everyone to vary 
the configurations of the spatial scheme of frontal direction, 
upright walk and left-right difference and 
thus to modify the “archetype” of space. Thus, 


through imagination, it is possible to imagine FIG.1 


what it is like not to stand but to crawl, i.e. not ATURE MORTE AVEC POMMES, 


to have any experience of the vertical. PAUL C EZA NNE 


In contrast to Cézanne’s times, today 
it is no longer possible to modify the way an 
object looks by means of a static or merely 
moving picture, but rather to make it look like an object: There are images that can be varied 
by the viewers themselves. Simulation images can be used to visually experience what it is 
like to steer a train or fly an airplane and thus conquer the vertical in a full sense. While the 
reception of these images was initially reserved for only a few people (such as pilots or scien- 
tists), in recent decades they have found their way into our homes and especially children’s 
rooms in the form of computer games. 

Although computer games are also means of play, they are first and foremost images 
of a special type: Their reception requires that they are not only viewed, but that they also 
interact with the visual phenomena via the incorporation of an avatar and the properties of 
it’s body. This applies primarily to those games that convey an interactive spatial image and 
thus — due to their interactivity or the real-time generated representation of the interaction 
result — provide a navigation scheme that takes over the function that the body has in real 
space orientation. Computer games, as simulation images, are thus media which, like the 
human body, structure and enable a specific spatial experience. Their basic feature is that of 
spatial navigation. 

Of course, the body as the archetype remains a reference point also for the avatar 
and the physical constitution of space the standard of comparison for a user of simula- 
tion images; but regardless of this, simulations can also provide completely foreign ways 
of experiencing things such as flying, which no one should know from their own physical 
experience. Flying or hovering have been used as perceptual modifications in cinema films, 


but for the first time the simulation image offers the possibility of navigating by oneself. 
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No film image, no matter how sophisticated, can 
create the impression in viewers that movement is 
induced by them. This is the principle of computer 
games. From a phenomenological point of view, 
those options are particularly important which are 
not in clear contrast to the extra-pictorial percep- 
tion, but which are rather variations or radicaliza- 
tions of the experience of the body. 

A radicalization is represented, for example, 
by the game ZORK from 1980, which represents 
the body schema as such in its essence. The game, 
which belongs to the group of so-called text adven- 
tures, is varied solely through the input of text 
and is thus already a borderline case of the image: 


the image itself contains no pictorial representa- 


tions of a space (fig. 2). In response to a description of a situation, players are asked to 


enter text, which consists mainly of control commands and subsequently triggers a spatial 
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movement, which in turn can be derived from the 
new description of the situation. Despite this text- 
based approach, the game is not about conversation 
with a virtual counterpart, but about navigating 
through a virtual space (fig. 3). 

In the computer game, the continuous 
movement as it is made possible by the medium 
of the avatar’s body (or in this case: location) is 
reduced to essential movements: In fact, topologi- 
cal navigation comes close to a phenomenological 
description of the human body, in so far as it is char- 
acterised by continuous movement, but, according 
to the phenomenology of the body, this also occurs 
between excellent directions along cardinal axes to 
the front or to the side. The only difference is that 
a movement can be repeated identically in the text 
adventure, whereas in the real world it can only be 
repeated differentially and results in similar but 
not identical movements of place. Only the statis- 
tical means of lifeworld movements would produce 
a space that can be called a “hodological space” 
by Kurt Lewin’s environmental psychology. The 
text-adventure ZORK therefore allows the space 
of decisions to be experienced as a pure hodological 
space. In this space it is no longer relevant how the 
users of the space get from one point to another, 
but only how they decide to go. At most, the aim 
of the game can be to traverse the labyrinth in the 
shortest possible way and to make the topology of 
the space itself the object of spatial action. 

Contrary to the hodo-topological spatial 
experience or movement, games of the First Person 


Shooter type largely approach the physical scheme 


of the human being, even though these also make a phenomenological reduction and, like 


text adventure, are based on spatial navigation possibilities that are topologically limited. 


‘They are different in terms of their visuality: Whereas in ZORK a black screen with bright 


letters could be seen, in games like DOOM a perspective view of space is given, i.e. players 
interact with an “egological” view (fig. 4). Accordingly, both genre names are derived from 
the subjective viewpoint that is the style of this spatial image, whose how consists in the 
central perspective and representational repre- 
sentation of its what. 

In contrast to a text adventure, a first- 
person shooter has a flowing or continuous 
spectrum of movement, so that in terms of 


navigation it is in one case an “analogue” 


F 


screen game, in the other a “digital” one, P| = ar 
BELDERER 

nical sense. Nelson Goodman in Languages n - — zi 

of Art from 1968 had already distinguished 


between “analog” and “digital” in the sense 


although both are digital games in the tech- 


of density and discontinuity of a representa- 


tion. This understanding can also be applied to 


interactive images: In the case of ZORK there 
are distinct, clearly distinct options (either FIG. 4 
“north” or “south” or “west” or “east”), in the ego-shooter, on the other hand, there is the DOOM (1993) 


continuous, grazing movement, in only vaguely defined directions. Accordingly, a compass 


is usually displayed in the game image or the game has its own map mode, with which or in 
which the primarily only approximately determinable directions can be subsequently con- 
cretised—and transferred into a “digital” determination. The experience of the “analogue” 
simulation picture ego-shooter can therefore come very close to the experience of space in 
the real world, as even people can only approach the points of the compass and GPS despite 
the aids they use, precisely because their being is physical. 

The difference between the “real life” situation and that in the game is therefore 
not, as in the case of the text adventure, the difference between “analogue” and “digital”, 
but rather the limitations and extensions of analogue spatial behaviour. To illustrate these 
variations two ego-shooters can be compared: One is the already mentioned DOOM from 
1993 and its successor QUAKE from 1996 and its sequels (fig. 5), both of which contributed 


significantly to the commercial establishment of the genre. 


FIG.5 
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Only in QUAKE was it possible to perform a movement in all three visible dimensions 
of space. In contrast to this, DOOM was designed in such a way that the user could only 
navigate with the avatar’s body or location along the horizontal axes of movement, even if 
the space was already three-dimensional or in depth perspective as a picture view. In this 
way, the image movement in DOOM was reduced to the two axes of the plane, whereby 
the primary lateral movement could be carried out in two ways: as a rotation around the 
vertical axis of the virtual body (clockwise or anti-clockwise to the right or left) and as a 
direct lateral movement. The latter in particular, as an excellent movement, is the secondary 
one in the game. This takes into account the fact that people also turn primarily in their 
everyday lives in order to take a different direction and do not move directly sideways to get 
to the right or left. Thus, the keys used in DOOM to move sideways are not located in the 
area of the other four keys on the input interface, where the relevant movement commands 
are given using the cursor keys [f], [|], [—] und [>]. The last two “mean” spatially then not 
“directly left” or “directly right” but the respective rotary movement. ‘The direct sideways 
movement, on the other hand, is performed using the [,] keys for “left” and [.] for “right”. 

All in all, the movement spectrum of DOOM is thus limited to navigation in the 
surface, which is paradoxical in that a surface has no extension and is therefore not space. 
At least this applies to the Euclidean conception of space. With the Non-Euclidean 
geometry, however, the paradox can be resolved: For example, with his Theorema Egregium 
the mathematician Carl F. Gauss in 1827 claimed, that the surface of the earth itself can 
certainly be regarded as a space, since it is not completely flat due to its curvature. In 
Euclidean geometry, the surface is a special case of space: a two-dimensional space. Thus, 
phenomenologically speaking, the spatiality of the human life-world is composed like an 
early ego-shooter game like DOOM in which the possibilities of movement of a planar 
being are simulated and only navigation in the plane is possible. The third dimension is only 
given here as a fiction: Even if the view of the room is given in “3D” and upright figures as 
well as walls and ceilings can be seen, the first person shooter cannot raise his gaze and look 
at the ceiling from the front or even look into the sky. 

The vertical change of view was only possible in QUAKE, which in turn required a 
change in the input: body and eye, which were still identical in DOOM insofar as they were 
moved at the same time, had to be controlled separately from now on: Thus in QUAKE the 
body of the avatar is still controlled with the left hand via the keys [W], [A], [S] and [D] 
in the topologically limited space of the surface, whereas the head is moved with the right 
hand via the computer mouse, insofar as the back and forth movement with the input device 
(which itself is moved in the surface) causes an up or down movement in the picture. 

For the movement, this Cartesian division of head and body or body and eye resulted 
in the fact that the lateral movement is no longer regulated by the keyboard alone, but 
primarily by the turning of the gaze, in so far as the previously marginalised evasive move- 
ment, which leads directly to the left or right, is now transferred to the central area of body 
control, but the rotational movements around the vertical axis now result from the combi- 
nation of the turning of the gaze (lateral movement of the computer mouse) and movement 
along the Z-axis of the picture space—i.e. in the dimension of its depth. In return, the 
evasive movement is raised to a tactical principle. These changes increase the demands on 
the players to act and spatially orient themselves in the image space of an ego-shooter. This 
means that even if the simulation supposedly approaches the physical scheme of the life- 
world experience, the game image varies considerably due to the interaction, thus forming 


not only an own style of representation but also an own spatial experience. 


THE AIS HAVE IT? 
HACKING INTO 
THE Al AVATAR 


DREAM’! 


1. TOWARDS THE Al 
AVATAR DREAM 


Avatars, artificial persons, or graphic placeholders for human 
beings are used in various functions in today’s cultures of 
digitality. Avatars range from cartoon figures — starting with 
“Clippy,” the famously annoying Microsoft Word paper- 
clip assistant whose googly eyes watch our moves on the 
screen — to virtual workforce employees, social partners, and 
programmed AI therapists. With avatars, we have to heed 
the disappearance of computers in society in the quest for digital 
humanity (Simanowski 2019: 3) by criticizing mere data- 
driven media and their cultural analytics (Manovich 2020) 
as models of AI avatar aesthetics. 

‘The avatar as a model of subjectivity has been described 
as a virtual proxy and representative of a real person (Little 
1999; cf. Gunkel 2010). Others focus on a prosthetic avatar 
as a puppet or homunculus double (Apter 2008) of agency 
in a technical milieu, including cybertherapy (Gerner 2020). 

The avatar dream (Fox Harrell and Lim 2017), when 


integrated with the two other culturally shared visions 


ALEXANDER GERNER 


IMAGE 1. 
CLIPPY: IN OFFICE VERSIONS 97 AND 2000, 
IF A USER TYPED “DEAR” AT THE BEGINNING 
OF ADOCUMENT, CLIPPY 
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Hil | am Clippy, your office 
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of future media of technological dreams using the computer and algorithms—the smart 


dream of ubiquitous quantitative total availability (Emrich/Roes 2011: 8-9) and the AI 


dream— becomes, in my view, the smart, ubiquitous AI avatar dream. 1 
Fox Harrell and Lim characterized the avatar dream in a twofold way: technical and 
experiential. Computationally created surrogates engage us using text descriptions in games 
or social media through virtual visual representations in virtual reality environments. The 
experiential dimension enables virtual surrogate selves to engage in immersive experiences 
beyond orthodox physical encounters (Fox Harrell/Lim 2017: 52). 
In this conception of the avatar dream, people utilize the computer as a chimera- 


creating tool to hack into their self-image. The avatar dream machine produces surrogates 
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and transforms each imaginary experiential frame virtually by allowing us to play who- 
ever we want to be. Thus, avatars might become part of a virtual identity. Fox Harrell and 
Lim (2017: 60) further argue that the avatar dream needs to be reimagined beyond mere 
techno-phenomenological otherness to take into account society, including biases and 
stereotypes and constraints to the achievement of social identity as experienced in physical 
and self-imaginations in virtual worlds. Suppose we do not heed the historical, social, and 
cultural constraints of human-made artifacts. In that case, we might not avoid system- 
embedded and user-embedded “box effects” —“the experiences of people that emerge from 
the failure of classification system (...) stereotypes, social biases, stigmas, discrimination, 
prejudice, racism, and sexism” (Fox Harrell/Lim 2017: 54)— that would render the avatar 
dream impossible. 

While the avatar dream is specifically related to personal self-image, the AI avatar 
dream goes beyond a mere computational representation of users. Beyond mere mechanical 
“learning” or “intelligence,” the AI avatar dream proposes AI avatars as creative machines 
(Rauterberg 2021). AI avatar dreams create other AI personas and professional specialists 
(e.g., therapists or consultants), such as embodied cognitive models, and a dream of another 
vision of humanity. in which the avatar is even part of 
a future self-generating art. This AI avatar dream goes 
in the direction of another artificial, virtual, or synthetic 
human: a form of self-superation, self-determination, 
and religious eternal self-salvation, with posthuman 
capacities, embodiment possibilities, and new modes of 


an extended human experience. Thus, virtual AI humans 


IMAGE 2. 
SCREENSHOT Al AVATAR WORK- 
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HTTPS://NEON.LIFE/NEWS/CES-2021-PRESS-RELEASE 


generate the future media ability to communicate in natural human language, to “learn,” 
“remember,” and “own” a personality as well as making decisions by taking actions with 
their bodies via a set of sensory systems. ‘The idea of virtual humans includes the ability to 
detect sensations, appraise sensation triggers, and respond to them. 

AI avatars act as a digital workforce and function as employees, such as the 
virtual worker AMELIA. AMELIA is a job-based, human-equivalent digital employee 
that is customizable for each service business, such as for Customer Care, IT, and HR 
services or multi-lingual digital banking. In the case of the Sterling National Bank, 
AMELIA—renamed “Skye”— provides human-like communication and collaboration 
with the bank’s contact center agents and in the case of the Netherlands-based IT Service 
“Centric Burgerzaken” AMELIA is used to provide conversational AI, available 24/7, for 
digital public services for local government organizations. 

AI avatars as algorithmically automatized workers are meant to enhance employee 


performance culture in VR scenarios within performance analytics. This development 


includes companies such as Talespin’s co-pilot virtual human training technology or 
customer assistance and UneeQ’s Digital Humans, defined as Al-powered, lifelike virtual 
beings. UneeQ’s Digital Humans are AI avatar workers that mimic human facial expres- 
sions, tone of voice, and body language in multimodal embodied forms of communica- 
tion. These features are more important than mere language-based verbal communication 
for customer service. The abilities of virtual humans include showing emotion and differ- 
ent moods, making plans, and achieving goals, ideally set by some “internal” motivation. 
Internal motivation in the sense of Artificial General Intelligence (AGI) could even be an 
internal avatar model with an external avatar body—with the AGI ability, in addition to 
reasoning and problem-solving, to mimic the capacity of imagination and creativity. Burden 
and Savin-Baden (2019: 13) have developed a matrix to analyze virtual humans’ traits 
on different spectra between self-aware and not self-aware, embodied and disembodied, 
humanoid and non-humanoid, natural-language and command-driven, autonomous and 
controlled, emotional and unemotional, personality-driven and impersonal, reasoning and 
unreasoning, learning and “unlearning” (cf. the EmoCOG architecture [Lin et al. 2011] or 
the OpenCOG architecture [Goertzel et al. 2014] in which attention-related “forgetting” 
and memory resource management is put forward [Burden/Savin-Baden 2019: 125]), and 
finally, imaginative and unimaginative. In a posthuman avatar case scenario, such as in 
Soul Machine’s 4th and 5th level of AI avatars, the aims are not only spatial context, as-if 
imagination, and as-if intentionality, but also creative machine behaviors based on “learned 
experience” and “agency” for making discoveries and setting new intentions, plans, and 
goals. Moreover, AI avatars in the future AI dream world gain the ability to train them- 
selves through interaction with humans and non-human systems. Finally, self-awareness 
and contextual understanding is supposed to emerge in independent digital, artificial 
persons with a strong semantic or contextual understanding of the Al avatar self’s actions to 
create non-linear storytelling. Nevertheless, AI artifacts that move, speak, reason, and show 


radical mimetism will inevitably face issues of animism. 


2. THE AIS HAVE IT? 
ON Al AVATARS 


2.1 “HIGH FIDELITY” AVATARS: COUNTERFEIT 
OF HUMAN GAZE OR THE WRONG KIND OF 
ANTHROPOMORPHISM? 


Al artifacts are AI systems that humans create for the purpose of radical mimesis: AI 
systems mimic actors who grant social faciality to machines in a way that seems human 
to observers. The AI avatar machine evokes movements of gaze and interest, as well as 
curiosity, and has to be critically assessed when reflecting on the topic of human or machine 
creativity. Coeckelbergh (2021) argues for a critical posthumanist point of view towards the 
anthropomorphism in technical objects that interact with humans. Should we then reject 
normative anti-anthropomorphism as nonsensical in social robotics and AI avatars? And 
still: we have to ask how we handle AI avatars not only as extensions of the self but as 
AI technology for human exploitation and data extraction (Crawford 2021), the cost of 
which must still be counted in its material, energetic and ecological (cf. Bender et al. 2021) 
aspects. Some may make a strong stance against AI avatars as simulation machines of not 
only intelligence but— foremost— human attributes such as creativity, autonomy, affectiv- 
ity, and for being “artifactors,” AI artifact systems that mimic human (like) actors, calling 
them a “counterfeit” (Pasquale 2020) of humanity. Therefore, the task of clearly separating 


AI systems from AI actors that mimic humans through anthropomorphic design stances 
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might seem a good idea for a policy option (Cf. EU 2021) that calls for a renewal of Asimov’s 
Three Laws of Robotics (see Pasquale 2020: 3-19). These new potential rules for AI would 
go beyond avoiding maleficence by impeding human substitution, human manipulation/ 


counterfeiting, an AI arms race, and non-identification of artificial systems. 


2.2 FROM Al CHILD AVATAR TO PLAYING GENERAL 
ARTIFICIAL INTELLIGENCE WITH A TOY CHILD 
MODEL: ON SOUL MACHINES’S AUTONOMOUS 
ARTIFACTOR ANIMATION 


The AI research of the company Sou/ Machines “started with a baby”, called “Baby X” (Soul 
Machines 2021). According to IBM (Soul Machines. IBM. n.d.) and its Watson assistant 
integrated into Soul Machines, the aim and business challenges are to build on the paradox- 
ical goal of empathic AI that has been staged as evolutionary human progress at the World 
Economic Forum in 2019 (Mantas 2019). The AI avatar model of Baby X plays interactively 
with the world around it, pragmatically making discoveries by manipulating things in the 
way we do. Animation stands at the center of Soul Machines’s business, which is inspired 
by the following questions: “What is the essence of animation? What if a character could 
autonomously animate itself and you could interact with it? How do you bring a digital 
character to life?” Baby X interacts with its surroundings by playing as if it were a child 
that learns, evolves, or “grows” its information base by testing the results of the games it 
plays; but does it actually rehearse and acquire reality? Soul Machines poses challenges of 


“problems to solve” that lie at the core of AI avatars as artifactors: 


How would we create biologically inspired artificial intelligence? And, build a digital 
consciousness to create affective computing that interprets and simulates human 
emotion, engaging autonomously? (Soul Machines 2020) 


Soul Machines’s Al avatar initially reminds us of an AI Tamagotchi (virtual pet), referring to 

the emotional annoyance of having to feed and care for the digital toy in how it is presented. 
However, the company aims to “make machines and Al as lifelike as possible,” envision- 
ing “humanlike AI that has flexible intelligence 
and a dynamic interface that can relate to people”: 
human-AI relations seem to change in the age 
of machine learning, having a clear roadmap of 
how to achieve the highest levels of “autonomous 
animation.” 

Soul Machines’s white paper (2020) dis- 
tinguishes six stages of autonomous animation, 
in which level 0 and level 1 are dedicated to 

actually existent simulated, actor-driven, 


pre-recorded video or motion capture in 


tions” in movie and games characters. On 
this level, avatars are supposed to be used 
as masks and puppets and heed movement 
notations of kinetic digital renderings in 
capturing performance art inside a motion 
capture imaginary (Karreman 2017) in creative industries, games, films, and contemporary 
dance. Avatar masks refer to a performer as a puppet master: the avatar mask can be seen 


as an initial new identity or as a mere puppet in an uncanny zone in between something 


and nothing. For Soul Machines (2020), on Level 1, basic pre-authored animation that is 
still actor-driven delivers pre-recorded movement based on simple triggers. The correspond- 
ing enabling technology would include the FAQ text-driven conversational database and 
pre-recorded voice content responses to create digital puppets. Levels 2 and 3 of “autonomous 
animation” would already use Natural Language Processing and “Dynamic Synthesized 
Human Behaviors” (Soul Machines 2020: 8), a “learning” capacity based as a solution on the 
Deep Fake level (level 2) or on level 3 with “[f]ull humanlike emotional responsiveness in 
facial animation including a conversational driven personality,” including on the voice level. 
With the selling of the idea of the 
higher-level autonomous AI avatar as part 
of the AI avatar dream machine industry, 
we should ask: Does an Al avatar assimilate 
otherness by radical mimetics to be used in 
game design and performative conventions 
for creating pervasive performances (Perez 
2016: 16) between acting and engagement? 
Do Al avatars follow the metaphorical model 
of Turing’s child machine to create and pro- 
vide “education” (Turing 2004: 460) to an 
AI child model such as Baby X or are they 
an AI avatar toy for playing 
around with artificial gen- IMAGE 4. 


eral intelligence, such as the SCREENSHOT OF A TRIALOGUE OF THREE MACHINE 
AT toy avatar model Kanzi COMMUNICATION AVATARS FROM SOUL MACHINES 


(Negarestani, 2018)? 
HTTPS:/WWW.YOUTUBE.COM/WATCH?V=4MCDPFKYLTS 
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2.3 CODEC AVATARS (CA): THE QUEST TO PASS 
FACEBOOK’S “EGO AND THE MOTHER TEST” 


As proximity and face-to-face encounters determine social relationships, the technological 
roadmap of VR and AR by Facebook’s Oculus Rift is heading towards overcoming distance 
and material barriers, as put forward by Tanaka, Nakanishi and Ishiguro (2014), who had 
shown that physical robot conferencing was superior to mere avatar chat. By virtual immer- 
sion of Codec Avatars, or enhanced Modular Codec Avatars (Chu et al. 2020)—which 
improve the robustness and expressiveness of traditional Codec Avatars— with holograms 
and VR/AR, Facebook aims at recreating and mimicking a sense of (artificial) VR tele- 
presence, which provides remote and immersive telecommunication through VR headsets. 
‘The training phase of the VR telepresence system in the first stage is done by capturing facial 
expressions of a user with a multi-view camera dome and a VR headset for face modeling. In 
the final phase a personalized face animation model is derived using these correspondences, 
while the real-time photo-realistic avatar is driven from the VR headset cameras. This social 
teleportation is able to share eye gaze and expressive faciality that would be almost indistin- 
guishable from the real-life presence of a person or object, even enhancing the spectrum of 
senses using a new artificial digital-media sense that could be called the digital immersive 
sense of foreshadowing proximity to an object or person. 

However, Mark Zuckerberg admits that a) not all material experience while 
“connecting people” will and can be virtualized and b) algorithmically modeling the mate- 
riality of touch and haptics is not easily done. Photorealistic avatar models for “high-fidelity 
social interaction” of the users’ faces render avatars with a “Deep Appearance Model for 
Face rendering” (Lombardi et al. 2018) “using non-linear, photorealistic full-face models 


of geometry and texture” (Richard et al. 2020: 1), overcoming the shortcomings of mere 
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geometric approaches due to the “non-linearities in texture-based tongue motions and lip 
articulation” (ibid.). The difficulties are related to dark untracked geometry inside the cavities 
of the mouth that must be emulated with a synthetic texture of the mouth. Facebook came 
up with the idea that the avatars should not only be acceptable but also that they should 
not create uncanny valley effects. When setting up an avatar, a second “Turing Test” of 
social presence for the Facebook Codec Avatar would be if the avatar is acceptable for your- 
self and “your mother,” (Tech@Meta 
2019). Thus, Facebook focuses on their 
codec avatars as an avatar dream of 
a high-fidelity replica of the gaze. 
Implicit in Facebook’s High Fidelity 
Avatar (Schwarz et al. 2020: 91) is the 
concept of high fidelity of Skarbez et 
al. (2017), who differentiate between 
a) physical morphological fidelity of 
looks inside the operational environ- 
ment, b) functional action fidelity of 
faciality of eye gaze or operational 
performance of the gaze in realistic 
movements and agency, and c) active 


perceptive fidelity. However, I ques- 


IMAGE 5. 
SCREENSHOT FACEBOOK CODEC AVATAR DEMON- 
STRATION VR TRAILERS AND CLIPS, YOUTUBE 
(JULY 2, 2020) “FACEBOOK’S PROTOTYPE PHOTO- 
REAL AVATARS NOW HAVE REALISTIC EYES” 
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tion if this hyperbolic-realistic “high 
fidelity” actually encompasses passive perception. 
What gives the face-to-face encounters a feeling of 
being together in the same space and experiencing a 
common “we”? Is the answer to this question related 
to the idea of being looked at by an other, who does not 


perform exactly as I expect? 


2.4 ERGOTIC COMMON-SENSE GESTURE-BASED 
Al AVATARS: TWENTY BILLION NEURON’S 
GESTURE SURROGATE AVATAR ASSISTANT MILLIE 
AND ITS Al APP FITNESS ALLY 


The former German/Canadian— and since its 2021 Qualcomm acquisition— US AI com- 
pany Twenty Billion Neurons (TwentyBN), works on computer vision and AI avatars: It 
teaches machines to perceive like humans. By developing the avatar “Millie” TwentyBN 
used a situated visual AI common-sense model via end-to-end learning from short video 
clips (Twenty Billion Neurons 2020): the “Supermodel.” This AI model is a Python-based, 
deep learning gesture-recognition model based on large-scale crowd-acting operations 
and has collected millions of short video clips that require no depth information, as the 
model is entirely trained on 2D video data. This gesture recognition model internalizes a 
visually simulated “common sense” of the world by identifying a wide range of fundamental 
human-object interactions and human body motions. 

The TwentyBN avatar is based on the AI SuperModel of computer analysis of col- 
lected crowd-acting, in which people in the recorded video snippets perform common-sense 
hand control gestures via different data sets. These include, for example, Jester V1, in which 
147 crowd workers performed 27 pre-defined hand gestures in front of a laptop camera or 
webcam (148,092 short clips of videos with different backgrounds, 3-sec length) and the 
“20BN-something-something V2 Dataset” inside the probability-guided labels to detect 


human common-sense actions by Al algorithms of machine vision. 


The neural network that offers the data feed to the avatar gesture simulation consists of 


short videos of mostly ergotic gestures sorted into common-sense pragmatic action classes 
(caption templates). These action classes are of a general “something [picking, moving, 
putting...] something” (Goyal et al. 2017: 5848) structure: AI avatars are based on com- 
mon-sense gesture training sets fed into AI algorithms. These AI vision algorithms use 
artificial neural nets and deep fake technology. The avatar Millie was introduced in the 
start-up phase of TwentyBN as an interactive AI avatar in-store shopping assistant, and its 
corresponding app “Fitness Ally,” a virtual avatar fitness trainer, was used to guide the user 


through a series of workouts and to present them with recorded and interactive training data 


for fitness improvement. 


Store Greeter Store Navigator Brand Ambassador Personal Stylist | M A G E 7. 
es SMARTPHONE APP 
IMAGE 6. “ 
l FITNESS ALLY “REAL 
SCREENSHOT TWENTYBN MILLIE’S FUNCTIONAL TIME INSTRUCTION 
APPLICATION AS “DIGITAL IN-STORE EXPERT. AND MOTIVATION” 


SEE: HTTPS://MEDIUM.COM/T WENT YBN/YOUR-DIGITAL-IN-STORE-EXPERT- 
FOR-EVERYTHING-D0865B82E27A. 


THE SLOGAN OF THE COMPANY IS “BREATHING LIFE INTO VIRTUAL BEINGS/ 
OUR HUMAN-CENTRIC Al TECHNOLOGY BRINGS SEEING AND SOCIABLE 
DIGITAL ASSISTANTS TO LIFE.” THE DATABASE IN 2017 CONSISTED OF MORE 
THAN 100,000 VIDEOS ACROSS 174 CLASSES; BY 2021 THE DATABASE HAD 
GROWN MORE THAN TENFOLD. 


HTTPS://FITNESSALLYAPP.COM 


IMAGE 8. 
SCREENSHOT OF THE 
Al FITNESS TRAINER 
ANIMATION FACE 
STACK DESIGN MADE 
FOR ALLY FITNESS BY 
FERGUI. 


(HT TPS://VIMEO.COM/543742486). 
(SEE: TBN 2020, DEC 14: HTTPS:// 
MEDIUM.COM/TWENTYBN/PUT- 
TING-THE-SKELETON-BACK-IN-THE- 
CLOSET-1E57A677C865). TWENTYBN 
IN 2020 ALSO LAUNCHED PART OF 
ITS TECHNOLOGY AS AN OPEN- 
SOURCE PLATFORM, SENSE, “A REAL- 
TIME ACTION RECOGNITION SYS- 
TEM,” OPEN-SOURCE INFERENCE ENGINE FOR NEURAL NETWORK ARCHITECTURES THAT 
TAKES AN RGB VIDEO STREAM AS INPUT AND TRANSFORMS IT INTO A CORRESPOND- 
ING STREAM OF LABELS IN REAL TIME. SENSE INCLUDES DAY-TO-DAY HUMAN ACTIONS 
(PICKING UP OBJECTS, DRINKING WATER, FIXING YOUR HAIR, ETC.), HAND GESTURES, 
AND FITNESS EXERCISES, AMONG OTHERS: HTTPS://GITHUB.COM/TWENT YBN/SENSE 
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Millie was created with Deep Learning training of initially one thousand actions; now the 
database of common-sense gestures and visual common-sense actions to feed this action 
recognition pool is far over a million. It contains an object detector, an action/motion detec- 
tor, a dialogue system, and a rule-engine for recognition and reaction to humans, which is 
used for Millie and was developed with the following aims (Kahn 2018): a) the immediate 
aim to build an interactive social sales assistant, gesture control systems for the car industry, 
and smart home devices b) the TwentyBN long-term aim to build full digital avatars with a 
designed personality to interact with people in various settings, including full digital social 
companion, exploring avatars that could even “help” teach children in schools or instruct 
adults in skills such as yoga or cooking, or an artificial officer such as New Zealand’s police 
artificial person “Ella,” developed by Soul Machines. 

Whether AI avatars will attain the depth and personality to serve not merely as 


trainers but actually as pedagogic teachers is an issue that remains an open question. 
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THE MASK 


DIETER MERSCH 


FIG. 1 
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Masks, like puppets, effigies, statues, and other artificial figures (fetishes, figurines, totems 
or idols) are one of the invariants of cultural history. They are found in all cultures, as the 
faces of animals or of the dead, as representations of hidden forces, meant to honor ancestors 
or give form to transcendental spirits and gods. Grimacing or serene, colorful, scary, static or 
almost life-like, in all cases they are a medium of an alterity, of something absent. Whatever 
is at their core, the mask, as a symbolic figuration, interacts with us as an “other Other.” 

Their form points obviously to a “faceness” (Gesichtiges) (Gombrich 1977). Masks 
clearly belong to what Gilles Deleuze and Félix Guattari have categorized as the body- 
head system. (Deleuze, Guattari 1987: 171) They point out that, on the one hand, the head 
can be separated from the body, yet, on the other, its “facenessness”—to not yet speak of 
“countenance” (Gesicht) or “face” (Antlitz)—is the clearest representation of head and body, 
because it provides their expression. There are of course heads without a countenance, cov- 
ered countenances as faces, even cavities that act /ike countenances, but the fact 
that we see them as such and attribute feelings to them only serves to further 
underline the exposed position of the countenance. Accordingly, masks of faces 
are found in all cultures as witnesses to a meeting with the self, as confronta- 
tions with an image of death, as representations of the strangeness of animals, 
as incorporations of anima or animus, as evocations of a transcendent or past 
world, as images of gods or as brushes with what Emmanuel Levinas has called 
the “enigma of the face” or its “numinousness.” (Lévinas 1979: 187ff.) 

The fecundity of the mask as an entity in cultural anthropology has often 
been remarked upon, as well as the fact that its many facets do not all belong 
in the same category, and yet they do belong to a common class (Lévi-Strauss 
1960). Masks are made of many different materials: feathers, fur, wood, clay, 
or metal, they are studded with gems and covered in fabric, they are painted 
and plain, carved from a single piece of wood or assembled from human or 

animal skull bones, they can also be 
ornate and adorned with a variety 


of emblems. There are halfmasks 


ONE OF THE OLDEST STONE MASKS EVER FOUND, that cover only the eyes, and masks 
EARLY STONE AGE NEAR ISRAEL, CA. 7000 BCE. Tee TEO NDER E a DON 


(MUSÉE BIBLE ET TERRE SAINTE, PARIS) 


the shoulder. There are masks that 


may only be used once and masks 


that are passed down from genera- 
tion to generation, with geographic variations. Finally, there are masks worn only to certain 
occasions or celebrations, and masks that are ubiquitous. Some display extreme expressions 
of sorrow or grieving, some are intricately modeled, and others are hollow forms with holes 


that reference eyes and mouth, as in one of the oldest archeological findings in Israel. 


FIG. 2 
AFRICAN MASKS 


PHOTO: MONIKA ZESSNIK © ETHNOLOGISCHES 
MUSEUM, STAATLICHE MUSEEN ZU BERLIN - 
PREUSSISCHER KULTURBESITZ. 

SEE HTTPS:/ARTINWORDS.DE/AFRIKANISCHE- 
KUNST/ 


FIG. 3 
AZTEC MASK 
OF THE SUN 
GOD TONATIUH, 
LONDON. 


SOURCE: AP 
HTTPS://WWW.WELT.DE/ 
KULTUR/ARTICLE4591746/ 
ALS-MONTEZUMA-SEINE- 
KUENFTIGE-BRAUT- 
HAEUTETE.HTML 


The symbolic meanings of masks are just as heterogeneous, as is their function within and 
outside of rituals and in practices of performative abreaction of trauma and other catastro- 
phes. They act as mediators between the beyond and here, between the ‘immortals’ and 
mortal humans, an echo of the past or the liminal space between presence and absence. 
But everywhere they stand for the intrusion of a strange force that must be placated; 
they push the boundaries of tradition, they question the present or negate the commu- 


nity while celebrating its allegorical destruction, its transgression and resurrection. As 


intruders from or heralds of another world, they spark a metamorphosis. 
They make a break in cultural life through the transformation that begins 
at their arrival. They renew connections that had become fragile or confirm 
contracts and political relationships. At the same time, they aid passages: the 
shamanic journey or the crossing into a new state such as initiation or death. 
They provide us with alien powers and cannot be controlled by us, instead 
they displace us to a “third” space beyond the facts of our birth, individuality, 


and social status. 


FIG. 4 
ANTIQUE THEATER MASKS 


HTTPS://WWW.PINTEREST.DE/ 
PIN/341358846746049638/ 


For these reasons, Roger Callois has insisted that “masks are the true social bond,” 


because they act not only as magical objects which attract other powers to themselves, they 


also take possession of the community to bring it together, renew it, or protect it from evil 


influences (Caillois 2001: 89). Yet at the same time masks reveal the fragility of social exis- 


tence; they summon the finitude of human life, the loneliness of death, and the insoluble 


historical disruptions of the conditio humana. Hence the mask refers not only to the “bond” 


(religio), but also to its dissolution and disintegration. It’s duplicity is thus three-fold: in view 


of our selves, of the Others and of the community as a whole. Hence the mask becomes a 


focal point; both the site of the true kowwvia (koinonia) as well as of the questioning thereof. 
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It is the appearance of an exception that has no place within the community, but neverthe- 
less both generates and problematizes community and its forms of “participation.” Excluded, 
it includes the possibility of community and a break with the same. 

As an artificial alterity, the mask evinces the ambiguity of the experience of the Other, 
of our dependency on others and their unbearableness. For communities are in no way simply 
there; they are permanently traversed by the rift of alterity which both allows them to be and 
divides them in equal measure. The mask embodies this conflict: the necessity and unavail- 
ability of the Other, their interruptions and resistance, their negation and foreignness and 
concurrently our unease regarding kowovia and its never-fulfilled promise. The mask is then 
the medium of an escape, a way of fleeing from the constraints of community while also prac- 
ticing the “other gaze” that allows us to see ourselves with distanced eyes. It creates a caesura, 
and holds up the mirror of alterity, with all of the word’s connotations, because the “truth” of 
the social state is the riddle of alterity as that which is always in the way and undefinable, that 
cannot be coopted or controlled by naming. Consequently, the mask denotes both the shiver 
of difference and at the same time the fascination for that enigma that religious studies scholar 
Rudolf Otto has linked to the “holy.” (Otto 2004) And in confrontation with the enigma, one 
is equally confronted with the ungraspable foundation, with the question why “the Seiende, 
that which is, is, rather than is not.” (Heidegger 1984: 2, 12; 1972: 53f.; 1965: 21) 


DUPLICATION 


All of the above helps to account for the extraordinary metaphysical depth of the mask, its 
quasi-philosophical “secret.” And yet it has not yet been fully defined. Hidden, it remains 
the unknown per se, holding an infinite diversity of cultural possibilities. Just as humans, 
according to Nietzsche, are “the still undetermined animals,” so do masks, as their desires, 
self-portrayals, transformations and divine foundation, remain indeterminable (Nietzsche 
2002: 56). No definition can capture it; the mask always proves to be more than it shows or 
purports to represent, more than it can symbolize, because of its many functions and prac- 
tices, even more than it promises to resolve through ritual performances. For this reason, it is 
more than just a medium, which it of course also is, because it breaks through the mediation, 
transverses its metamorphoses and realizes its potential in a never-ending round of transfor- 


mations. As Claude Lévi-Strauss clarified in his study of masks: 


It would be misleading to imagine ...that a mask and more generally a sculpture or 
a painting may be interpreted each for itself according to what it represents or to 
the aesthetic or ritual use for which it is destined. ... On the contrary, A mask does 
not exist in isolation; it supposes other real or potential masks always by its side. 
(Lévi-Strauss 1982: 144) 


This means that masks are constantly “becoming.” Yet there is more, for we are looking not 
only at relationships of inclusion and exclusion and their movement, but at a network of 
artifacts, their history, and their futurity. Thus when Caillois calls the mask “the true social 
bond,” that is not the whole of the matter, whether we are looking at processes of identity 
creation, communication, the localization of social roles or the attestation of sovereignty, 
because experiences of destruction, death and renewal are equally important. From the 
beginning, the mask is therefore situated in an “other” place, a beyond, an u-fopos, that 
evades every attempt at determination. Masks therefore touch simultaneously upon the pos- 
sibility and the impossibility of the social bond, upon our mortality and our immortality, 
and upon the legitimacy and illegitimacy of our identity. In the end, they tear down the 
difference between being and appearance or between truth and falsehood, and always have 


a share in the chronic precarity of cultural meaning. 


Where then does their infinite perpetuation come from, their evolution and the desire for 
the replication of forms, the desire to create further “figures” and “faces” besides ourselves, 
ambiguous creatures on the line between the oppositions that constitute sociality and con- 
sciousness, as if it were important to continually shift and unsettle that border and create 
doubt? Do they satisfy our wish to ourselves be creators, to constantly duplicate ourselves 
between veiling and exposition, between disguising and discovering? Is the mask above 
all an GyyeAoc, an angelos, a messenger of the paradoxicality that swings from passivity to 
activity, between the given actuality (Ge-gebenheit) of the world and the contingency of the 
social bond, always threatened by collapse?! Does it herald the primary unrealizability of 
community and its culture, raising its demonic head wherever it is accomplished, mocking 
its hopes and economies? ‘Then perhaps the dynamic creation of multitudes of artificial 
Others—the boundless multiplication of not only masks but also sculptures, dolls, mario- 
nettes, and images and maybe also todays animated figures, including robots and artificial 
intelligence—is a panicked sublimation of our mortality and an attempt to liberate ourselves 
from it, to banish the death wish and in so doing separate ourselves from the Other and the 
intrusion of the autonomy and corporeality of others. Don’t all of these figures have some- 
thing of a corpse in them, insofar as, as Slavoj Zizek provocatively stated, “the only good 
neighbor is a dead neighbor,” since only the dead body and ossified face is under our control 
and at our bidding, reconciling us with its otherness (Zizek 2005). 

In that case, ownership, displacement, and power are the strongest drives behind our 
desire to create artificial Others that we can burden with all of the ambivalence, hate, and 
guilty conscience that resides within the social bond and its gaping wound. Consequently, 
masks transpose the negativity of affect by transforming the dialectics of social dependency 
and resistance into a second, abstract “face” that haunts us equally with its threat and with 
its laughter, putting fear into us. If so, we must also admit that the mask cannot be an object 
of libidinous desire, for it is neither loved nor rejected, neither recognized nor despised. 
Rather it transfers our feelings by, standing in for the real Other, acting as a foreigner or an 
intruder, submitting itself to a delusion that is in equal parts destructive and power-hungry. 
Put another way, this means that we need duplication in order to again and again find our 
way in the labyrinth of our own incomprehensibility as well as the unworkability (deseuvre- 
ment) of the social bond, In short, the fragile nature of our existence Nancy 2010; 1990; 1991). 


DUPLICATE AND 
DOPPELGANGER 


To counter this fragility, the mask is linked to the simultaneity of covering and discovering, 
of veil and revelation, to the extent that that which is hidden is displayed all the more. But 
this dialectic is given only when the masquerade is interpreted symbolically and as part of 
the order of representation. Yet the play of opposites has more depth and more branches, it is 
more ambiguous and complicated than it seems at first glance. For the mask does not cover 
one face only to offer us another with which we can present ourselves on the social stage. 
It also does not denote the travesty of the face (Antlitz) under which our true countenance 
is revealed. Rather it exposes the concealment and then masks the exposure and vice versa, 
disguisung while at the same time uncovering the covering, so that we are always dealing 
with a play of double negatives. 

This negation should not however be confused with a contradiction in discourse, for 
it is an iconic “obliterative” negative. Obliterative discourse always obscures an Other that is 
not questioned in its presence but remains chronically undefined. It is a practice of negation 
that connects to a practice of deletion, disguise, highlighting, and striking—not for nothing 
did the mask as tpd0wnov (prosopon), the tragic actor’s cloak, and 60opvog (cothurnus) belong 


1 


Weihe (2004) also views the mask 
as a contradictory form, albeit 
from the perspective of systems 
theory. 
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FIG.5 


together in Ancient Greek theatre—that, following Sacha Sosno and Emmanuel Levinas, 
can be called “obliteration.” (Lévinas 2019) The mask is the epitome of such obliteration; as 
a superficial, hollowed form it erases its wearer and simultaneously makes them into some- 
thing incomprehensible that, through a collection of holes in place of the eyes, mouth and 
nose, speaks of bodily orifices and insinuates an unknown interior. It follows that it is not 
the mask itself that is mysterious, but the way in which it is used, which produces a secret by 
presenting something visible that at the same moment flips into something invisible, touch- 
ing upon the mystery of life itself. Made 
of cavities and openings, the mask hints 
at a space behind it, a dimension of an 
Other kind, that is similar to that which 
we discover in all life and in every creature 
that shares with us an indelible bond of 
elementary relationship. 

This links the mask—in contrast to 
machines and their mathematical simula- 
tions such as avatars, which are nothing 
more than black boxes—with another 
deeply-rooted cultural figure, the doppel- 
ginger. The doppelganger, too, confronts 

us with the instability of our 
identity and hence also with 


the countenance as an expres- 


SACHA SOSNO, AUTOPORTRAIT 1997, sion of SUPPOSE EEE 
GRAPHIC PRI NT, SILKSCREEN. and as the form of its attributes. 


SEE HTTPS://WWW.PICASSOMIO.COM/SACHA-SOSNO/43559.HTML 


The duplicate separates us from 


ourselves, it is both the same 


FIG. 6 Ck Moa EN 
GIORGIO DE CHIRICO. 
THE TWO MASKS, 1926 


and foreign; at the center of the 

face is creates a rift, a difference. 
The mask has the same effect, except that it puts one countenance upon the 
other, dispelling its certainty and allowing a glimpse of an infinite space 
“behind” it, just as the doppelganger transforms the unequivocalness of 
identity into a ghost. 

Over the course of the Middle Ages, the phantasm of the doppel- 
gänger became a spectral object that not only portends something beyond 
appearances that lasts after death, but more importantly suggests that 
we—simultaneously “divine” and “bestial” or “dividual” and “individ- 
ual”—can not be one with ourselves. For if we believe that life is satu- 
rated with a transcendence and a divine “gift,” then, just as the individual 
means the iöi@rng, the idiotis who does not have a part of the whole and 
shuts himself out of the community, an insoluble contradiction is constitu- 
tive for our existence. Woven into a world of superstition, deception, and 
confusion, doppelgängers appear as messengers of fate, as a sign of or in 
anticipation of future events that tear open the gap, the rift, and make the 
contradiction manifest. 

The mask is cut from the same cloth, like the mask, the doppel- 
gänger obeys the genuine duplicity of ovpBdAAEw, symbdllein, to throw 
together, and d1aBpdAAEw, diaballein, to throw apart, or of connecting and 
separating, as well as of the circle of life caught between order and chaos. That is why its 
appearance makes us uneasy. The mask can therefore be seen as a preeminent object of 
ambivalence, as a magical point, at which the phenomenon of the doppelgänger also 
crystallizes, which is why Hans Belting (2017: 32ff.) situates its origin in the death cult, in 


the effigy as the simultaneity of decay and preservation. The frozen expression, the 


petrification of its features receives a second covering as a momento, so that the mask 
becomes a continuation and an outliving, confronting the épnpepos (ephemeros). We are, as 
Pindar wrote, “creatures of a day,” doomed to die and okıäg Övap &vOpwnos (skias onar 
anthropos), only the “dream of a shadow” that needs to give form to memory to transcend its 
own painful dissolution.? 

Seen in this way, the mask also promises continuity of the dead in life, their resitua- 
tion at the site of the duplicate persona, caught in constant transformation and participating 
in both kingdoms: the living, social world and the persona as the actual image of the dead 
person. While this primary meaning as a statuary sign of those whose “faces had been 
lost” is mostly temporal, a second, derivative meaning is suggested that must be understood 
spatially, namely the presentation of an absent power (Belting 2017: 34). The mask thus 
becomes a figure of power, an emblem of the presence of the absent ruler, while also belong- 
ing to the order of ephemerality and its concurrent precarity. For just as human fragility 
demands permanent symbolic restitution for more than a day, so does power. Hence a 
symbol of power is inscribed in the mask that reconstitutes political power at the site of its 
absentia. Its appearance is then like a memorial that has no rea/ power, but all the more imag- 
inary power, whereby the imaginary is more powerful than the manifest, for it has already 


internalized power as an idea. 


THE PUBLIC COUNTENANCE 
AND THE ABSENTIA OF THE 
FACE 


In both cases, however, the mask is linked to duplication and so associated with the figure 
of repetition, of which images of death and power are merely the two most distant poles. 
Belting also reminds us that face and mask intersect, and that we pay great attention to 
shaping our social faces as the surface of expression and proof of our personality. For that 
reason, Gilles Deleuze and Felix Guattari have suspended all representational logic and 
melded mask and face: “The mask does not hide the face, it is the face. ... everything that 
is public is so by virtue of the face.” (Deleuze and Guattari 1987: 115) That means we must 
read the expressive face not only—as in physiognomy—as a “signifying system,” but also 
attribute theatrical passion to it through the pose that is part of theater’s exquisite repertoire. 
‘Theater also becomes an exemplary medium for the description of sociality (ibid.). Masks, 
whether of the everyday or the sublime, are always theatrical, just as in theater the face is 
subject to the tireless economies of adaptation, where it throws itself into the public eye and 
tries to draw all eyes to itself. 

Rainer Maria Rilke drew a caricature of their sometimes dismal efforts in his novel, 
The Notebooks of Malte Laurids Brigge: 


There are many people, but even more faces, since everyone has several. There are 
people who wear a face for years, and of course it wears away, gets dirty, wears 
in the creases, stretching like gloves ... Those are thrifty, simple people, they don’t 
change their face, they don’t even have it cleaned. ... Other people change their faces 
over uncannily quickly, and wear them out. At first they think they have enough of 
them to last them forever, but hardly have they reached forty, they're on the last 
one. ... They aren't used to looking after faces; their last wears through in a week, 
has holes in it, and in many places it’s as thin as paper; and then gradually the base 
layer starts to show through, the non-face, and they go around wearing that. 

(Rilke 2016: 4-5) 


2 


Pindar: “Creatures of a day. 

What is someone? What is no one? 
Man is the dream of a shadow.” 
(Pythian 8: 95). See Frankel 
(1946): 132-133. Eprpepog, in 
early Greek literature, does not 
mean “creature of one day, short- 
lived” but “subject to the (chang- 
ing) day, variable,” and the term 
implied that along with the shifts 
and changes of a man’s life, his 
outlook and character fluctuate; 
thus man is unsubstantial and a 
mere “shadow in a dream.” 
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FIG. 7 


The public face wears out, but there is no final ground “behind” it, no authentic face to 
counter the faces that are put on, as is said of the death mask. Instead, there can be nothing 
under the mask but more masks. 

‘That is why there are only masks and no faces, as the mask behind the mask behind 
the mask behind the mask behind the mask etc. exposes an abyss and finally ends at the cast 
of the dead person, whose expression is as distorted as the first mask. We are confronted 
with the concept of ‘nothing’ as found in Jacques Derrida, for example in the inconspicuous 
“a” in différance, assuming the mask does not signify some- 
thing extant but only itself and its never-ending proliferation. 
(Derrida 1982) Masks are not pulled over the face in order to 
hide it, rather they reproduce the face as a supplement. Yet the 
question remains whether the chain of supplements exposes 
a trace of that which does not allow itself to be masked. For 


Deleuze, the answer is clear: 


Repetition is truly that which disguises itself in constitut- 
ing itself, that which constitutes itself only by disguising 
itself. It is not underneath the mask, but is formed from 
one mask to another as though from one distinctive point 
to another, from one privileged instant to another with 
and within the variations. The masks do not hide any- 
thing except other masks. (Deleuze 2001: 17) 


This however implies that all masks are related and only 
function when they are constantly repeated, shifted, and 
transformed, and also, as Deleuze continues, that repetition 
can only be “that which disguises itself.” Correspondingly— 
expanding on Nietzsche’s ideas in “On Truth and Lies in a 
Non-moral Sense”— the differentiation between truth and 

falsehood or between authenticity and 

fraud is obsolete (Nietzsche1993). Noth- 


TONI SAULNIER (1926-1968) A PARIS, RUE JACOB, ing can be repeated before the repetition 


LE DINER DES COLLECTIONNEURS DE TETES, 1966 that it refers to or that could be referenced 


PHOTOMECHANICAL REPRODUCTION FROM 
“LART NÈGRE” PARIS MATCH, 21 MAY 1966 Having said that, the key difference is not 


from it, because “the mask is the true 
subject of repetition.” (Deleuze 2001: 17) 


between genuineness (Eigentlichkeit) and 
masquerade, but between mask/counte- 
nance on the one hand and ‘face’, in Levinas’s sense, on the other, for the “face” is not the true 
countenance, but the invisibility of alterity in the countenance—that moment that makes a face 
human and forces a truly ethical turn. We see countenances and observe them, identify expres- 
sions, mood, and mimicry, but we meet faces, albeit only when we allow ourselves to be touched 
by them, so that the mode of experience is not aio®notc, afsthesis, but response and responsibility. 

The distinction that thus comes to light is homologous to the differentiation between the 
two types of relating, namely through identification and so through perception and interpretation, 
including the origin in naming and the circulation of the sign, and on the other hand through 
passivity in the form of accepting and acknowledging, within which is always immanent both the 
fear and the respect for that which we are not. Beneath the mask is no aödevrng (authenzes) that 
must be discovered, but the “trace” of the alter, both behind the masks of others and in our own 
mask-likeness. Put another way, there is a categorical difference between the mask and alterity 
that is subsumed neither in the differentiation between truth and falsehood nor in that between 
authenticity and pretense, but which oscillates between identity and non-identity, for alterity is 


neither true nor false, just as the mask can not be primary or derivative. 


SIMILARITY AND 
DISSIMILARITY 


Importantly, masks are not iconic; they are not primarily images, but plastic objects whose 


most conspicuous characteristic is perhaps their negativity in the form of those cavities that 


we have delineated as obliterative. They not only gape open in lieu of eyes and mouth—the 


places through which voice and gaze and thus the “trace of the Other” escape—but they 


also form cavities that allow us to animate them with magical possession. Even if every mask 


does not necessarily need a wearer, still they are not moved 


by avtóuatov (automaton), but by something that, while 


ecstatic, communicates with us and appears to share with us 


of the same mystery. For that reason, all masks incorporate 


something Other into themselves, indicating the relation- 


ship of all creatures, the inability to grasp existence, and 


the power of drives that resist control, as well as the unruly 


longing of life and its unavoidable relationship to death. 


FIG. 8 


EMIL NOLDE, MASKEN II, 1920, 
OIL PAINT ON CANVAS, 74 x 88,8 CM. 


That means, however, in complete opposition to technical artifacts, whose vectors 


aim at hypermimesis and consequently at a similarity that is almost sameness, for the mask, 


it is dissimilarity that is important. It is avöno1og (andmoios), for if one compares mask and 


face, the former is like a distortion or grimace of the latter, in which a faceness begins to 


appear and is rejected in the same breath. That which is dissimilar is however constitutive 


for every similarity, and not the other way around, its difference also makes reflexive access 


possible. Mathematical models in contrast aim at the technological simulation of identity: 


their téA0¢ (telos) is homomorphism, whose main principle is structural sameness. They aim 


at mimicry, at being deceptively real, the perfect imitation. Here we see one of the charac- 


teristic misunderstandings of a technological era that has more than its share of misunder- 


standings, for its proliferation of artificial figures stems from a misguided supposition about 


people and their humanum, 
which has no use for copies or 
for the radiance of illusionism, 
but always only for what is 
heterogeneous and finite, and 
for the resulting self-restraint. 
This does not, finally, denote 
a lack, but potentia, for only 


“mortal aims befit mortals.” 
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CREATING 
MY DIGITAL 
SELF 


STELLA SPEZIALI 


INTRODUCTION 


In 2019, in the context of building up the Immersive Arts Space, an art/tech lab at the Zurich 
University of the Arts, I was asked to investigate how to create a digital twin from a real, 
existing person. It was about essential questions like, figuring out how to scan him or her 
with different techniques or clean up the data, handle the rigging, and master the anima- 
tion. The goal was to create a digital twin of that person as a final output. I decided to boldly 
take on the challenge, not knowing at the time the steep learning curve I would be on. [am 
not afraid to experiment with things I am not familiar with. I am also in constant pursuit of 
the latest technical innovations to understand and deepen my skills. Back then, I only had 
basic understandings of how to build a hyper-realistic digital human. Nevertheless, having 
myself available at all times as material to work with, I decided to create my digital twin. 
Needless to mention that making a digital human is not a simple task. A series of 
carefully calculated steps needs to be executed. Otherwise, you may find yourself redoing 
and correcting the same steps several times. Each one, no matter how small, will affect the 
final result. So, before I started creating my twin, I made a list of criteria that I wanted to 


follow: 


° The objective was to create a coherent self-presentation as close to identical with one’s 
physical self as possible, in order to achieve a realistic, aesthetically looking digital 
human. 

e An important aspect was to use a combination of low and high-tech solutions and 
equipment, so as to keep a lid on costs. 

* The outcome was planned to be animated in real-time by motion capture technology 


and integrated into virtual environments. 


Given the continuous technological progress in creating virtual humans, an essential general 
focus is photo-realism (Bartl et al. 2021). At the same time, there is still an ongoing debate 
about whether realistic-looking virtual humans are prone to facilitate the uncanny valley 
effect, which describes the phenomenon that close-to-real looking artificial humans may 
strike us as eerie (Mori et al. 2012). The manifold approaches for realistic 3D-reconstruction 


vary mainly in their technical complexity and the resulting costs involved. These two factors 


influence the degree of realism that can be achieved. So far, expensive multi-camera rigs, 
based on high-quality image sensors —as used as in professional light stages — achieve the 
best quality. However, alternate approaches, based on more affordable consumer hardware, 
for instance, single 2D photo cameras or smartphones, are becoming more elaborate and 
thus more popular. Most of these low-cost approaches share the vision to make the inex- 
pensive production of digital alter egos possible for everyone without a complex hardware 
setup (Bartl et al. 2021). 

Together with my fellow researchers, Florian Bruggisser, Patxi Aguirre and Tobias 
Baumann, we focused on consumer-level hardware and invested much time in simplifying 
the complexity of the processes. We also had the opportunity to be mentored by an expert 
in the industry.’ 


CREATING MY TWIN 


I used a combination of low-cost solutions, such as using a single DSLR camera to capture 
multiple views of my face and body. I also applied a more complex and expensive hand- 
held device to produce detailed scans of facial expressions and body parts such as teeth 
and hands. ‘The first method uses a photogrammetric reconstruction [Fig. 1] and provides 
a rather rough 3D reconstruction of my face and body that needs further processing and 
optimisation. The advantage, however, is that it produces a basic surface texture that can be 
used for the 3D model later on. 


FIG. 1 
The second method leads to precise and very PHOTOGRAMMETRY PROCESS WITH THE SOFTWARE 
detailed reconstructions. In fact, for the peal ITY CAPTURE. ORIGINAL PHOTO, POINT CLOUD 


facial expressions and hands, I used an Ein- 


Sean Bro, whichieraplays: a modular eure: RECONSTRUCTION, 3D RECONSTRUCTION WITHOUT 
ture light projection (it generates a striped TEXTURE, 3D RECONSTRUCTION WITH TEXTURE. 


pattern) that produces the 3D reconstruc- 
tion of the scanned surface [Fig. 2]. With its 


much faster processing, this scanning device can provide a series of distinctly different facial 
expressions [Fig. 3] in a relatively short period. It also manages to capture wrinkles and 


other morphological details of the skin [Fig. 4]. 


pä 


Matthias Wittmann, who is a 


š : i member of the Digital Human 
the 3Shape TRIOS scanner to generate the 3D reconstruction. Both methods, in combina- research group at Digital 


For scanning my teeth, I asked my dentist to make me an intraoral scan, and he used 


Domain, reviewed our progress 
and advised on how to master 
by standard Extended Reality (XR) simulations and game engines such as Unreal or Unity. complex processes. 


tion, generate high-quality virtual humans ready to be processed, animated, and rendered 
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FIG. 2 


HANDHELD SCANNING. 


FIG. 3 
SCANNING RESULTS OF MULTIPLE 
EXPRESSIONS AFTER CLEAN-UP. 


FIG. 4 
MORPHOLOGICAL DETAILS OF THE SKIN. 


With each iterative 3D model of myself, we compared the result in an immersive virtual 
environment. For this, we imported the reconstructed 3D model into the digital universe of 
a game engine and compared the appearance of my authentic self with the digitally recon- 
structed copy. My virtual variants are perceived consistent in terms of similarity to the 
original, human-likeness and uncanniness, even though we perceived them differently in 
terms of realism. As I have a more intimate awareness of my body than my colleagues, I have 
the impression that my virtual body is less realistic than my fellow researchers think it is. 
Strangely, I perceive it as plastic and synthetic. Consequently, with each additional version 
of my digital twin, I feel pressure to create an increasingly perfect version of how I look in 
real life [Fig. 5, Fig. 6]. 


GETTING TO KNOW MY TWIN 


Once the digital twin exists and is manipulable, a new set of questions 
arises. How do I perceive and approach my digital self-presentation? 
Furthermore, will this new self-presentation affect my self-conception? 

Motion capture technology offers the full-body tracking of my 
movements and the transfer of these movements to the digital twin. 
This enables consistency between one’s physical body and the digital 
double. An additional and similar transfer can take place regarding 


facial expressions by means of performance capture. The importance 


of exact correspondence of bodily and facial movements is essential 
for constructing and experiencing a self-representation. The body FIG.5 

itself becomes the immediate and sole instrument to control the DIGITAL TWIN 1.0 
digital twin and thus provides an intimate connection between MADE IN COLLABORATION WITH MY COLLEAGUES. 
one’s physical body and one’s digital self (Freeman/Maloney 


2021). 


P 


FIG. 6 


SIDE BY SIDE COMPARISON OF ME AND MY DIGITAL TWIN 2.0 


MADE IN COLLABORATION WITH MY COLLEAGUES AT THE IMMERSIVE ARTS SPACE, ZURICH 
UNIVERSITY OF THE ARTS. 


The perceived realism of virtual humans depends just as much on their movements and 
behaviour as it does on their photo-realistic appearance. Until now, this project focused on 
appearance rather than behavioural realism. Nevertheless, this does not exclude a further 
investigation once the digital twin is completed and usable in interactive real-time situations. 

Since the digital body can be fairly manipulated and customised, this can raise 
questions regarding identity, race and gender. By doing this, I have the opportunity to 


deviate entirely from my real-world identity, projecting myself in a way that looks nothing 
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FIG. 7 
AVATAR MADE BY ME 
WITH READY PLAYER ME. 


FIG. 8 
AVATAR MADE BY ME 
WITH FACE HERO. 


FIG. 9 
METAHUMAN MADE 
BY ME WITH UNREAL 
ENGINE METAHUMAN 
CREATOR. 


like Ido in real life. Digital spaces afford the experimentation and projection of entirely new 
identities, but what does this mean for me as a human being? What does it mean for others 
that know me, but interact with my digital twin? The types of relationships we have with 
digital representations of ourselves can vary widely. Moreover, since the way we represent 
ourselves is often nuanced and complicated, it does not stay static. (Fisher, 2021a). Some 
people choose to create a fictional character and have an avatar that looks nothing like they 
do. Others will make an idealised version of themselves, whereas others like me willtry and 


make the most true-to-life version they can (Fisher 2021b). 


SHIFTING IDENTITIES AND 
GENDER EXPRESSIONS 


I have also tried to recreate myself on different platforms, from the stylised versions [Fig. 7, 
Fig. 8] to a hyper-realistic version [Fig. 9]. 

A stylised avatar offers some level of abstraction and idealisation as well as different 
levels of body ownership and periodically shifting appearances. At the same time, it also 
enables one to “experience different forms of embodiment— including human, nonhu- 
man and perhaps somewhere in-between” (Virtualis 2020). The sense of authenticity and 
the multiplicity of digital identities surface when the avatar is given a personality, unique 
behaviour, intentions and style. When we create avatars, we never truly and completely 
capture ourselves. However, nuanced presentations emerge as intertwining relationships 
among different dimensions of identity emerge, such as appearance, gender (female, male, 
binary and non-normative), race, ethnicity and age (Freeman and Maloney 2021). 

Yet, there are no precise rules on how we choose to portray ourselves in the digital 
world. The main constraints are given by the platforms, the knowledge and capacity we 
have, and of course, by a great deal of resilience and motivation to experiment with those 
intangible differences between one’s physical self and the digital counterpart. 

When it comes to using virtual humans and avatars, the Proteus effect is a prominent 
research topic. It describes the phenomenon that the avatar appearance can influence users’ 
attitudes and behaviour based on stereotypical beliefs (Bartl et al. 2021). Indeed, selective 
self-presentation and performance occur in people’s behaviour, perception and cognition, 
since they conform to their avatar and the mental stereotypical characteristics and per- 
sonality the avatar represent. This representation of self may be a different social identity, 
one without the limitations of the physical world, which leads to a seemingly unrestricted 


expression of self (Freeman and Maloney 2021). 


SEEING MY TWIN DAMAGED 
AND DEFORMED 


Looking at all the different avatars I have created, I believe my hyper-realistic digital twin 
represents me the most accurately. This is probably because of the amount of work I have 
dedicated to it, including all those frustrating moments in the creation process when I finally 
realised that other techniques would work better than the one I chose previously, including 
those instants of embarrassing hilarity when you see your digital twin being awkwardly 
deformed. 

This is likely to happen during the animation process, particularly when my digital 


twin gets moved and animated by male colleagues trying to solve technical issues. In that 


process, mistakes can happen, and then I find myself seeing my digital twin completely 
distorted [Fig. 10], maybe also with the wrong face texture [Fig. 11], almost like a monster. 
However, the fascinating thing is that there is an adequate distance between her and me, 
mainly on the empathic level. Even if I perceive a particular violence in the facial deforma- 
tion or skin aesthetics, this violence is not perceived as directed to me, but aimed at some 
other entity. It may be due to my experience of playing characters in realistic 
games, where I have the feeling that the 3D model of myself is empty, is dis- 
similar enough from me, from my human person, to remove almost entirely 
that uncomfortable connotation that accompanies the damaged digital body. 
Yet strangely enough, there is still a lingering feeling of unease created by the 


mingling of humour with violence. 


CONCLUSION 


I am intrigued by the possibility of experiencing my everyday self in a novel way 
and exploring undiscovered potential. Then again, having concretely experienced 
my highly realistic digital twin makes me more sensitive about my identity. I am 
actually a little sick of always seeing my face and body in digital form. 


Yet it has encouraged me to immerse myself in the idea of a different 


body, inhabiting unfamiliar gender and ethnic spaces. FIG. 10 


By continuing this research, I would like to investigate diverse DIGITAL TWIN WITH 
new practices, phenomena, and interaction consequences surrounding DISTORTED FACE 


the self-presentation mechanisms in virtual environments. I believe it 


is increasingly important to understand how people represent them- 


selves digitally and how virtual environments might influence their —————————— rl 
behaviours, especially by outlining the potential benefits and opportunities in terms of REFERENCES 
ren f identity and gender matters. This might include raisin tion ut an Em 
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THE BIRTH OFA 
DIGITAL ACTOR 


THABO BEELER, DEREK BRADLEY 


Digital humans have become of central relevance in current film production. Out of the 
five top grossing films released to date (2021), four make use of digital humans of some 
sort. The outlier, Titanic, was shot long before digital humans were practical, back in 1997. 
Since then, technology has leaped forward, making photoreal digital humans a reality. Still, 
the creation of such digital characters poses a formidable challenge and is oftentimes con- 
sidered to be the Holy Grail of visual effects. The quality bar is tremendously high, since we 
humans have been conditioned by evolution to scrutinize human faces and even the slightest 
imperfection can destroy the illusion and trigger rejection of the character— the infamous 
uncanny valley phenomenon as introduced by Masahiro Mori (2012 [1970]). A digital actor 
is typically born by first scanning his or her likeness, then building up a digital model, which 


can then be animated or retargeted to a separate digital character. 


SCANNING THE LIKENESS 


In order to cross this uncanny valley, the industry relies on reproducing reality as faithfully 
as humanly possible. To create effects such as explosions, water or destruction, physically 
based simulation has been invented. To emulate the way light interacts with materials we 
rely on physically based rendering. To create digital characters, studios rely on capturing real 


humans at different stages. While this holds in general for the entire human, we will focus 


on the most challenging aspect in the remainder of this article—the human face. 


FIG. 1 


MULTIVIEW RECONSTRUCTION SETUP AND EXAMPLE 3D GEOMETRY. 


© DISNEY. 


First, the shape and appearance of a human face will be acquired using scanning tech- 
nologies. Shape refers to the 3D geometric structure of the face, where appearance denotes 
its color and how it interacts with light. Interestingly, the delineation between shape and 
appearance is scale-dependent, and small scale structures such as skin detail may sometimes 
be considered geometry or modeled as part of skin reflectance. To acquire the shape, an 
actor is typically captured in a photogrammetry or videogrammetry setup (Beeler/Bickel/ 
Sumner/Beardsley/Gross 2010). Such setups consist of multiple cameras, ranging anywhere 
from two to two-hundred, and are accompanied by reconstruction software that converts the 
multi-view face images to a 3D face model. At their core, these algorithms all function simi- 
larly as they rely on the fact that points at different distances from the cameras will project to 
different locations inside the captured images. Hence, identifying 
corresponding features across views allows to triangulate them in 
space to recover their 3D position. While there are other methods 
that employ more advanced hardware for 3D scanning, such as 
structured light (Weise/Li/Van Gool/Pauly 2009) or depth sensors 
(Li/Yu/Ye/Bregler 2013), nowadays passive photogrammetry has 
become the method of choice since it is highly accurate and requires 
relatively inexpensive hardware. 

As indicated, shape alone is not sufficient to render realistic 
images since it lacks the appearance information. Skin exhibits 
very intricate appearance properties due to its physical structure, 
which makes both acquisition and reproduction very challenging. 
A fraction of the incident illumination is reflected off the top layer 
of the skin, the so-called stratum corneum, causing the highlights 
called specular reflections. How much light is reflected 


depends on the incoming light direction relative to the 


© Cross-Polarized Camera 
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local orientation of the skin surface. These reflections 
preserve the color of the light source, since the rays do 
not interact with the lower skin layers at all. The rest of 
the incident light traverses the stratum corneum enter- 
ing lower skin layers, specifically epidermis and dermis. 
While it travels through this volume, bouncing from 
molecule to molecule it gradually changes its color as 
skin absorbs certain parts of the light spectrum more 
than others. At some point it will exit the skin again, 
now appearing flesh colored. ‘The exact color depends on 
the parts of the skin it travelled through and the distance 
it covers. This property is called subsurface scattering 
and is the reason why our skin appears soft and translu- 
cent. To measure these reflectance properties, typically 
studios rely on a lightstage, which is a large device that 
allows to illuminate the face from a large number of 
light directions (up to several hundred) in a controlled 
way (Debevec/Hawkins/Tchou/Duiker/Sarokin/Sagar 
2000). By observing the varying appearance at a specific 
point on the face lit from multiple of these light sources, 
it is possible to estimate the reflectance properties of the 


face, such as albedo (color without shading) or specular 


FIG. 2 
COMBINED FACIAL GEOMETRY 

AND APPEARANCE CAPTURE FROM 
RIVIERE, GOTARDO, BRADLEY, GHOSH, 
BEELER 2020. A) CAPTURE SETUP, 

B) EIGHT PARALLEL-POLARIZED 
VIEWS, C) FOUR CROSS-POLARIZED 
VIEWS, D) RECONSTRUCTED GEOM- 
ETRY. BOTTOM: EXAMPLE RECON- 
STRUCTED APPEARANCE, INCLUDING 
(FROM LEFT TO RIGHT) ORIGINAL 
IMAGE, RECONSTRUCTED RENDER, 
DIFFUSE ALBEDO, SPECULAR INTEN- 
SITY, NORMALS, HIGH QUALITY 
GEOMETRY, ANOTHER RENDER UNDER 
DIFFERENT LIGHTING. o pisney. 


attenuation. The need to be captured in two different setups—one for shape and model build- 
ing (described next), and one for reflectance—is obviously suboptimal and recent research has 
suggested a way to add appearance acquisition to the well-established videogrammetry set- 
ups, yielding a one-stop-shop to digitize the likeness of the human face (Gotardo/Riviere/ 
Bradley/Ghosh/Beeler, 2018; Riviere/Gotardo/Bradley/Ghosh/Beeler 2020). 
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FIG. 3 


MODEL BUILDING 


The scanning technologies introduced in the previous section are employed to capture a 
human actor performing a number of expressions. How many and which expressions depend 
on the requirement of the model building stage (also known as rigging). The typical approach 
is to model the human face holistically using a linear blend-shape model—a strategy that 
represents all facial expressions as a linear combination of a set of base expressions (Lewis/ 
Anjyo/Rhee/Pighin/Deng 2014). After scanning the base expressions and converting them 
into acommon mathematical representation where vertices correspond between shapes, new 
expressions or entire performances can be created and animated by manipulating the weights 


of the base shapes in the linear combination. These models are extremely robust and intuitive 


as 


for artists, yet they require to capture a large 


EXAMPLE FACIAL EXPRESSIONS THAT ARE SCANNED number of base shapes in order to provide 
USING MEDUSA TO FORM AN ACTOR MODEL. opisney,  “frdent flexibility and expressiveness. For 


example, a model consisting of two shapes, 
say neutral and smile, will be able to express 
these two shapes and any shape in-between, 
such as a half smile, but it will not be able to express, for example, a frown. Hence people 
rely on sampling the human expression space as systematically as possible, oftentimes fol- 
lowing the Facial Action Coding System or FACS system as introduced by Ekman and Friesen 
(1978). This system isolates facial motion into atomic “Action Units” (AU) as a function of 


the underlying muscle structure. For example a smile may be produced as a combination of 


AU 6 (cheek raiser), AU 14 (dimpler), AU 15 (lip corner depressor), AU 17 (chin raiser), and 
AU 24 (lip pressor), among others (Schmidt/Cohn 2001). As a result, global blend-shape 
facial rigs typically require the acquisition of hundreds of base shapes. The challenge of this 
system in practice is, however, that humans are typically not capable of consciously activat- 
ing individual muscle groups. Furthermore, since most scanning systems are seated, effects 
such as secondary dynamics due to motion are not captured and hence cannot be modelled. 
Nevertheless, building an actor model from a fixed number of scanned base shapes is still 
the method of choice for most productions today. 

To overcome some of the limitations of global blend-shape models, recent research 
has proposed more flexible /oca/ face models, which model the deformation of faces in 
smaller local regions (‘Tena/de la Torre/Matthews 2011; Wu/Bradley/Gross/Beeler 2016). 
This allows to obtain more variation in the combined facial expressions from fewer scanned 
base shapes, but this approach 
comes at the cost of robustness, 
as inconsistent behavior across 
different local regions can result 
in uncanny shapes that no lon- 
ger resemble faces. Such a prob- 
lem, however, can be alleviated 
by considering spatial regu- 
larization and the global ana- 
tomical structure of the face, 
in particular modeling the ex- 
pression-specific skin thickness 
between the skin surface and 
the bones (e.g. skull and mandi- 
ble) (Wu/Bradley/Gross/Beeler 
2016). Such an anatomical local 
face model has become a new 


option for digital face modeling FIG. 4 


in recent film productions. LOCAL ANATOMICAL FACE MODEL FROM WU, BRADLEY, 
GROSS, BEELER 2016, SHOWING THE FACE GEOMETRY, THE 
DISTRIBUTION OF SKIN PATCHES, AND THE UNDERLYING 


BONE STRUCTURES. opisney. 


PERFORMANCE ANIMATION 


Once an actor’s face model is built, it can be animated by artists using the rig to create novel 
performances for the digital avatar. However, obtaining precisely accurate facial motion is 
extremely difficult by hand, and oftentimes digital characters that look realistic in still life, 
suddenly look uncanny when they move or speak. For this reason, it is customary to also 
capture performance animation from the real actors, and map this onto the digital character. 

Performance capture has a long history, and by now most people are familiar with the 
idea of motion capture suits for body tracking, where actors wear tight fitted clothing covered 
in retro-reflective balls that can be tracked using infrared cameras. This notion of “sparse” 
mo-cap is then translated into full body character animation. For the case of facial perfor- 
mances, similar sparse motion capture has been achieved by painting small black or colored 


dots on the actor’s face and tracking them in video. This provides a low-resolution repre- 
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sentation of the facial performance, which can be used to approximately drive the higher- 
resolution actor model. A major drawback of marker-based performance capture is the need 
to place the markers on the actor’s face, which not only requires time, but also requires 
the makeup artist to place the markers in exactly the same locations from day to day over 
the course of a production. Furthermore, the result is only a rough approximation of the 
facial motion, since accurate information can only be obtained at the marker locations. For 
this reason, follow up research focused on markerless “dense” performance capture of faces 
(Beeler/Hahn/Badley/Bickel/Beardsley/Gotsman/Sumner/Gross 2011), which uses high 
resolution synchronized video cameras and tracks the face at the skin pore level, yielding up 
to a million accurate point locations tracked for each frame of a performance. Such a system, 
called Medusa, was developed by DisneyResearch|Studios and was awarded an Academy 
Science and Technology Oscar in 2019.7 
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FIG.5 
SEVERAL FRAMES FROM AN ANYMA PERFOR- Markerless face perlormanee capture systems a 
M AN C E CA PT U R E, S H OWI N € T H E | N P U T W | T H Medusa typically require a well-controlled studio 


WIREFRAME RESULT OVERLAID (TOP) AND 


setup, consisting of many cameras, bright uniform 


illumination, and the actor is usually required to stay 


STABILIZED GEOMETRY WITH RIGID SKULL seated with their head in a fixed position—all in an 


MOTION REMOVED (BOTTOM). opisney. 


1 


https://studios.disneyresearch. 


com/medusa/ (02.02.2022) 


https://studios.disneyresearch. 


com/anyma/ (02.02.2022) 


effort to make the complex problem tractable. In order 
to allow a more “unencumbered” performance, more 
recent research has investigated model-based perfor- 
mance capture, where an actor-specific motion prior 
is used during the reconstruction process (eg. Tewari, Zollöfer, Kim, Garrido, Bernard, 
Perez, Christian 2017). This prior is often just the actor model (or rig) built from the facial 
scans as described above. Optimizing for the model parameters directly to match the video 
(so-called ‘analysis-by-synthesis’) is a convenient way to recover the performance, since the 
output is already in the required format (e.g. mapped onto the actor rig). Using the prior also 
constrains the set of possible facial shapes that can be reconstructed, and this allows to lessen 
the physical burden, allowing performance capture using less cameras (even just 1), and in 
less constrained environments like outdoors. Along these lines is the Anyma performance 
capture system,’ developed by DisneyResearch|Studios, which uses the anatomical local 
face model mentioned earlier (Wu/Bradley/Gross/Beeler 2016) as the prior. This approach 
provides among the highest quality 3D facial performances and has been used in several 


recent film productions. 


RETARGETING 


There are several practical scenarios for creating digital actors. One very common one is 
when there is a need for a digital character that does not exist in the real world, but can still 
be “performed” by a real actor (e.g. Marvel’s Hulk performed by Mark Ruffalo). In such 
scenarios, a digital version of the actor’s performance is generated, and then transferred to 
the ultimate digital character seen on screen—a process known as performance retargeting 
(Ribera/Zell/Lewis/Noh/Botsch 2017). 

When retargeting a human actor performance to a separate digital character, it is com- 


mon practice to undergo most of the process of scanning and model building for the live actor, 


even though they may never appear on screen. This helps to reduce the problem to a purely 


FIG. 6 
SEVERAL FRAMES OF A PERFORMANCE RETARGETED FROM THE SOURCE 
CHARACTER (TOP) TO A TARGET CHARACTER (BOTTOM). © pisney. 


3D geometric one—how to map the deformation of one 3D face (the source) onto another 
(the target). The problem is challenging because oftentimes the source and target faces have 
different proportions, different dynamics and could even have very different bone and muscle 
structures (for example, a human mapped to a dragon). Several different methods for per- 
formance retargeting have been developed over the years. The most straightforward approach 
is to build complete, identical blend-shape models for both the source and target characters, 
and once the captured performance of the source actor is modeled by a time series of blend 
weights, the same weights can simply be used on the target blend-shape model to obtain the 
retargeted performance. The obvious downside of this approach is the time and effort required 
to build two complete and expressive blend-shape rigs, carefully crafted such that the 
expressions artistically correspond in the desired way. But once the models are constructed, 
large numbers of performance sequences can be readily converted from the source character 
to the target in no time, and thus this approach is popular in production environments. 
When constructing multiple blend-shape models is impractical or otherwise unat- 
tractive, an alternative approach is to retarget the surface motion directly. If the source 


and target characters do not differ greatly in proportions, the time-varying per-vertex 
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— 
3 https://www.fxguide.com/ 
fxfeatured/how-virtual- 
production-worked-on-set- 
of-the-lion-king/ (02.02.2022) 
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CONCLUSION 


In this article we discuss the birth of a digital actor, involving several important steps like 
scanning the likeness, model building, performance animation and retargeting. Each of 
these steps has seen tremendous growth in technology and research over the last decades. 
With every new innovation, we get closer and closer to being able to cross the uncanny 
valley, and perhaps some digital avatars already have. 

Despite the progress, researchers continue to push the limits of “actor to avatar” tech- 
nology. One current drawback is that most methods for high quality performance capture 
require a lot of processing time and are thus executed offline after the performance video 
has been recorded. In this era of real-time virtual production where film-makers are eager 
to direct digital characters in digital environments in real-time,’ the next big challenge will 
be production-quality face capture at real-time frame rates, from a single camera directly 
on a film set. While some of the latest capture methods do achieve real-time performance 
(Feng/Feng/Black/Bolkart 2021), they do not yet reach the quality of offline approaches. 
Still, with the speed that technology is developing in this field, we likely won't have to wait 


much longer. 
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HUMANIZING 
VIRTUAL PEOPLE 


MATTHIAS WITTMANN 


When we talk about virtual humans in real-time space, we often refer to them as Avatars. 
However an Avatar is only a digital representation of an actual existing person, driven by 
that person. Autonomous virtual humans on the other hand are virtual beings that are con- 
trolled by “themselves” through artificial intelligence (AI). 

Currently there is a lot of talk about AI and we constantly hear buzzwords like 
“Machine Learning”, “Neural Networks” or “Deep Learning”. But what is AI? One defi- 
nition is the theory of developing computer systems able to do tasks that normally require 
human intelligence. 

A lot of Al is being used for understanding processes and then using that knowledge 
on other merely similar (but not the same) processes. Facial recognition is a good example: 
Once an AI system has been “trained” on enough human faces it will be able to recognize 
a human face on a picture, even if it hasn’t seen that particular face before. This is possible 
because during training the AI system is learning more abstract information about these 
images. For example: humans do have two eyes, a nose between but below the eyes, one 
mouth centered under the nose ... and so on. So what happens if a Zyklop is in the picture? 
The cool thing with AT’ is that they never just give you results based on “yes” or “no”. They 
return probabilities. And they do that very well. So when an AI sees a Zyklop it will most 
likely return something like a 65% probability that it is a human face. This behavior by itself 
feels very human. Of course you can also get wrong results from an AI. Its skills to interpret 
unknown information depends on the complexity of their “training“. 

So why don’t we already have perfectly AI driven robots or virtual humans if AT’ are 
so great? That’s simply because reading and interpreting is easier than producing. Reading 
an emotion from a person’s body language or face or even from the content of their spoken 
words is much easier than coming up with an appropriate emotional reaction: The right 
body or facial pose, motion or gesture. And that is simply because the training of reading is 
easier than the training of doing. Reading emotions versus creating emotions. For me it is 
all about creating emotions. This article will illustrate where I was in that process in 2019 
and how I got there. 

After receiving my diploma from the Filmakademie Baden Wirttemberg in 
Ludwigsburg, Germany, I started in 2000 as a character animator in the movie industry. 
In 2008 I became the animation lead of “The Curious Case of Benjamin Button”. Work- 
ing a lot with the modeling department I became something like an expert for human 
facial anatomy, facial muscle movement and skin micro motion. Over the next few years 
I was mostly involved with digital human animation and development like “Tron Legacy”, 
“Virtual Tupac” (the Coachella hologram) and “Maleficent”. 

While working in animation I always spent some time on coding tools. Not only 
because it was helpful, but also because it was a lot of fun! I am not a schooled program- 


mer, so writing code for me is like playing a game. If it works, I won. And as opposed to 
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30CM VIRTUAL ASHTON KUTCHER 


animating, coding gives you a clear result. Either it works, or not. Sure, you can optimize 
code to work more efficiently (which is fun too), but the results stay the same. Animation on 
the other hand is always open for interpretation. 

In 2010 I started using Unity at home to create little games for my iPhone. And 
while I mentioned before that coding means fun to me, coding for real time engines is really 
exciting. Four years later in 2014 a friend of mine asked me if I would be interested in join- 
ing a company to develop interactive virtual humans for Augmented Reality (AR). That is 


when I switched from film productions to realtime development. 


VIRTUAL HUMANS IN AR 


The first project I was involved in was a 30cm virtual Ashton Kutcher presenting a new 
tablet right in front of you on your desk. You could watch him in AR through your own 
tablet. I added a “look at us” logic to make him, well, look at us whenever he talked to us. 
My knowledge from high-end film productions came in handy even in the low-res world 
of mobile real time. It was important to me that the way he looked and moved would feel 
natural, which was tricky considering he consisted of only about five polygons. 

The next AR project utilizing a tablet as a window to the augmented world was a life 
size stewardess promoting an airline. She did not have to talk but be friendly and invite us 
to the airline’s booth at a convention. Aside from a ‘look at us’ mechanism I added reactions 
to her based on our distance. When we were in reasonable range she would beckon us and 
if we got closer I let her smile. 

Both of these AR projects felt pretty successful from an emotional stand- 
point. But now I became curious about how far one could go with Virtual 


Reality (VR). For a start one could use better graphics since serious VR sys- 


tems were still connected to a PC. And how immersive would it feel sharing 

the same three dimensional space without just a tablet as a window? 
First I tested it on an old film asset: a human head 
with a few really well modeled shapes for eyelid 
deformation. It was the first time for me using 
Unreal Engine and I programmed the eye and 
head rotation in a way that the head would look in 
my direction in VR. The results felt mind blowing 
realistic. Of course, it did not look photorealistic, 
however the presence of seemingly another being 


was undeniable. 


THE SESSION 


This confirmed that one could go much fur- 


ther with the given technology and in 2015 A LIFE SIZE STEWARDESS 
for my next project “The Session”, I devel- PROMOTING AN AIRLINE 


oped a full-size virtual human in VR: An 


old professor. This time I let the character 
speak to the player whose reactions and decisions determined how the story proceeded. 

To establish communication between player and virtual human I decided to go with 
gesture recognition. The professor was able to read the player’s nodding and head shaking 
as “Yes” and “No”. All AI features were developed based on fuzzy logical decision trees. The 


whole system was supposed to (technically) fake human behavior. 


The professor was able to memorize objects and could also forget them if he would not see 
them for a while. Just as humans favor certain features in the images they perceive of the 
world around them (motion, contrasts, vanishing lines) this vision AI was supposed to do 
the same. Although I did not get to implement the other two elements, the Professor was 
able to notice motion, especially if it was fast or sudden. While he was not able to notice 
static objects in his peripheral vision, he did notice them when 

they moved. He also was aware of the fact if the player was looking THE SESSION 
at him, but only if he was looking at the player at that moment. 


I did not create a locomotion logic at this point, meaning the Professor would not be able to 


avoid a player and could potentially walk right through them. But he nearly never did since 
the story was designed to predict where the player actually was. the AI only walked into 
spots where it knew the player could not be. For example: The Professor told the player to 
go to a pillar in the room. If the player did not go there the AI would not change location. 
However, if the player went over to the pillar the AI would know that the player just freed 
up space and the Professor could move without the danger of intersecting with the player. 

Aside from the actual game part I also created a test-level where the user could just 
“hang out” with the professor and take a closer look at him. While he would not move or 
engage in a conversation with the player, he still would react to their presence. 

The professor would look at them when they entered his field of view, he would nod 
or shake his head when the player did and would react confused and even annoyed when the 
player got too close. These reactions felt very personal giving the player the impression that 


this being had real feelings. 


MAN AT A BUS STOP 


In 2018 I got the chance to design and create an even more advanced interactive virtual 
human in VR. The project was called “Man at a Bus Stop”. I had several goals this time. One 
was having a way more complex environment, to showcase that a complex interactive virtual 
human in VR could be used in an equally complex environment without overwhelming the 
computational power of a decent, yet not overly hi-end PC. Other goals were adding hair, 
voice recognition and developing an emotion engine. Also, I decided to update the user 
interface giving the player hand representations this time. 

The user experience was based on two principles: firstly the character AI should be 
as humanlike as possible not only in terms of look and behavior, but also in abilities. This 
AI should not be all-knowing. The human-Al should only see, hear and feel what a human 
would. No eyes on the back of the head... Secondly the user should not have to “learn” any- 
thing. Anything they would be able to do should come naturally to them. 
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Like “The Session” before, “Man at a Bus Stop” is based on a real actor, Tom Fitzpatrick. 
We started as we would for film productions. We did an extensive body scan. This time we 
experimented with a new method using a mobile hand scanner solution. It was based on 
photogrammetry but would be moved around the subject. The actor had to stay still for a 
longer time than in a classical volume, however its flexibility was a huge advantage. 

The face was scanned on a light-stage, a system originally invented by Paul Debevec 
at the Institute of Creative Technologies (ICT) at USC. This process creates geometry that 
is so precise, it shows every single pore of the actor’s skin. The resolution of such scans is 
very high, 7.5 million polygons in this case. While one would not use the high-resolution 


geometry for real time, one still gets the necessary information 


MAN ATA BUS STOP |S BASED ON for creating highly detailed displacement maps. Displacement 
AREAL ACTOR TOM FITZPATRICK, mapscangive a low-res geometry the look of a much higher res 


one, while needing less resources. For that we remodeled the 
head with a much lower poly-count of 30.000. ‘Then we used a 
modeling tool that compared the two heads. This produced a black and white image that 


represented the delta between high res and low-res geometry: the displacement map. 


DIFFIUSE 


BODY SCAN AND FACE SCAN Aside from high-res 
geometry the Light- 


stage scan produces 
also other very important maps that are needed for shader creation like defuse, specular and 
normal maps. To create realistic looking skin, we utilized Unreal Engines ability for sub 
surface scattering (SSS) shader models. Skin reacts strangely to light. It absorbs a lot of it 
and reflects it in a more diffuse way. Without an SSS shader it is very difficult to create this 
look. That is why digital humans looked like “plastic” for a long time. In real time graphics 
they often still do. The main reason is that even though this kind of shaders do exist now, at 
least in some game engines, they need more rendering power, which usually results in lower 


frame rates. 


Since you cannot just scan hair, it had to be created separately in Maya using XGen. Luckily 
Tom had very short hair, so we did not have to deal with dynamics. Nevertheless, in the end 
the poly count of just the hair was ten times higher than that of the rest of the face, including 
eyes and teeth. 


PLUS ... 30,000 TRIANGLES x AND ANOTHER 325,000 
(Eves, TEETH, ...ETC) NNNNA l ... FOR HAIR 


At this point we should talk briefly about the Uncanny Valley, the effect that we subcon- 
sciously accept simplified representations of humans more than representations that are only 
“almost perfect”. The question is, how to overcome the uncanny valley? What makes it 
uncanny? ‘There is no simple answer. Anything can pull you out of the illusion, look, motion 
and even the voice. 

Let us assume we start from a really good 3D model of a person which has already 
decent shaders. Most of the time the trouble starts with the eyes. ‘They are, as some may say, 
the gateway to the soul. Poorly designed eyes can be disturbing even when the character is 
not moving at all. Motion is another factor. The most perfect looking eyes will still break the 
illusion if they do not move and deform the surrounding skin like we are used to. And then 
there is the rest of the face. A human face is very complex and to recreate it with computer 
graphics is only accepted by the audience if it moves as a whole. If you only pose the mouth 
to the spoken words but do not move the rest of the face accordingly, the illusion falls apart. 
Everything in the face, and the whole body for that matter, is interconnected. There are 
specific ways how our head moves while we are talking, and there is a way how our body 
moves when our head is moving, and vice versa. If you break any of these relationships, you 
run into danger of falling deep into the uncanny valley. 

Once things are moving, deformation and therefore volume preservation becomes a 
factor. When pulling a rubber band, it becomes thinner. The same basically happens in the 
face, you pucker your lips, the cheeks are getting stretched and flatten against your teeth. 
Moving parts of a face and not taking care of volume preservation will be subconsciously 
noticed very quickly. There are even more possibilities to break the illusion, yet these belong 
to the most important ones. If not taken care of properly you get reactions like “Something 
is off, I can’t tell you exactly what it is, but it’s not real”. 

To make sure to have proper facial motion and deformation I went with DI4D to 
do a facial performance capture. A volumetric scan like that returns a deforming facial 
geometry of the complete performance. Such performance clips are very large and it was 
clear that I would not have the bandwidth in VR to run it on high frame rates. So we used 
an algorithm from an old siggraph paper to convert the complete performance into a small 
number of animated blend shapes. However, blend shapes and animation created through a 


process like that can’t be edited. The data looks too random. We still managed to enhance 


the rig to allow for adding emotions and moving his eyes interactively. — 
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At that time high quality scans like these were not possible to be captured simultaneously 
with body performance. So we did a separate body motion capture session of the same 
actions. It is not an easy task for an actor to reproduce the same action twice, once sitting in 
a chair and once moving on stage. Ideally the body capture should happen before the face 
capture session and both should happen on the same day. That way it is easier for the actor 
to understand and remember the context 
when he has to sit still. And believe me, 
it is a lot to ask from an actor to perform 
‘lively’ without moving their body. 

Now we had two separate cap- 
tures. To connect the facial and the 
body performance, I spent some time in 
editing. It was important that his facial 
rhythm would fit to his body motion. 
One could not just line up the voice 
recordings of the two captures, since 
length and rhythm was always slightly 
different. So the key was to find the most 


important ‘beat’ in his performance, and 


line it up to that. 


FACIAL AND BODY PERFORMANCE an 


dealt with separately. A performance scan moves the skin, but the hair 


geometry does not know about that. So we had to design a special joint setup 
that would move the hair together with the head. Aside from the scalp that 
also included the eyebrows and eyelashes. 

In order to make a character feel naturalistic within an experience like “Man at a Bus 
Stop”, multiple layers ofinteractivity come to play. First there is the actual story line. To give 
the illusion of choice, the story has to branch from time to time. Branching often feels better 
for the user, but also creates more work. Key to an efficient branching design is that it leads 
back to the main track as soon 
as possible. Else the amount 
of options will increase expo- 
nentially. 

Aside from this obvious 
kind of interactivity I added 
layers of behavior to make 
Tom feel more like a real 
human. For once there are 
actions the character can 


do simultaneously with the 


main performance. Turning 


towards us when we are not 


A SPECIAL JOINT SETUP THAT WOULD MOVE right in front of him is one of them, or leaning away from 
THE HAIR TOGETHER WITH THE HEAD us when we get too close. These motions have to be well 


balanced. The eyes, head and body turn towards us with 


individually tuned speed, acceleration and precision. 
There are also more subtle actions to keep him “alive”, that react interactively too. His 
breathing and eye blinking frequency change based on his current mood. So does his mimic. 
Like before with the professor for “The Session” it was a no-brainer to add emotions 
to the character again. Though this time the emotion engine was more elaborate. Rather 
than changing the face and pose just based on e.g. distance, Tom would now remember his 


own emotions and at times even holding a grudge for a while. That indeed was a very power- 


— ful update that added a lot to his “humanlike” feel. 
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ADDING VIRTUAL HANDS 


In terms of game play and controlling the action, dealing with the interaction of humans and 
virtual humans is challenging. Since players in VR are sharing the space with the character 
it is harder to predict what they may do. Some people new to VR, are often overwhelmed 
and have a hard time focusing on the story. While others are quickly bored if there is not 
enough “action”. Some players try to break the system, others do not move at all. 

So how to deal with that situation? First of all you need to have a basic understanding 
of user psychology. What attracts them? What bores them? How can you gain their atten- 
tion? The easiest way is to follow the concept of reward and punishment: The character asks 
the user to do something. If the player fulfills the request something entertaining happens. 
When the user moves too far away from that scene just fade to black. (Don’t fade to black, 
it is mean!) 

Adding virtual hands as user representation to “Man at a Bus Stop” was well received. 
To avoid breaking the immersion the hands were designed with situational awareness, 
mostly based on the distance to Tom and the area one would reach out to. That way it was 
not necessary for the user to learn anything. When someone reached out to Tom the hand 
naturally posed in a way that made sense. An interesting result was that even though the 
hands clearly changed shape while moving, everybody just accepted this behavior as normal. 

Having said this, I mentioned above that players are hard to control in VR. Give 
them VR hands and see what happens. The reward and punishment principle becomes 
blurry. The AI reaction to touch is rewarding, but it can also disturb the story, for example 
when touching his face while he is speaking. Tom reacts clearly unhappy when you poke 
his nose and while some users are feeling 
bad about that, others are motivated by it. 
Most players treated Tom “respectfully”, 
yet some twirled their hand inside his 
head, only to see if they could break him. 

Interesting was that almost nobody 
would actually try to walk through these 
virtual humans, neither the “Professor”, 
nor Tom. There seems to be a natural 
barrier when VR space and characters 
become too real. 

When players used the system as 
intended, by playing the story and listen- 
ing to Tom, “Man at a Bus Stop” became 
very rewarding. The fairly hi-end design 
and the sophisticated Emotion Engine 
made it feel very intimate. 


Tom came already very close 


to being a very human-like THE FAIRLY HI-END DESIGN AND THE SOPHISTICATED 
virtual being. Who knows EMOTION ENGINE MADE IT FEEL VERY INTIMATE. 


what lies in the future ... 
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DOUBLE TROUBLE. 
DIGITAL AVATARS 
ON STAGE 


CHRISTIAN ISELI 


INTRODUCTION 


‘The interplay of technologies from the film and gaming industries enables digital avatars 
to climb onto the stage. However, both media have prerequisites that affect the realization 
of digital characters. In film, it is primarily realism, both in terms of photorealistic sur- 
face textures arising from the depictive tradition and in terms of precise human motion, 
resulting from the tradition of animated films, wherein humanized movement contributes 
significantly to the audience’s ability to empathize with a character. The realistic concepts 
lead to the ideal of lifelike digital humans as well as of animated movements and facial 
expressions that are confusingly human-like, regardless of whether characters are stylized 
toons, realistic animals or human characters. In computer games, on the other hand, the 
focus is on interactivity, and this results in the priority of ultrashort response times, to 
which other aspects such as graphic style and the impression of realism in texture and move- 
ment are subordinated. The same priority also applies to the performing arts, where the 
overriding premise is the experience of a live event with physically present actors, singers 
or dancers. Digital avatars on stage increase the demand for real-time performance even 
further, because not only the rendering of the characters, but also the animation based on 
performance capture needs a final output in real time. In distinction to this, movements 
in games are preproduced. Motion capture data is first recorded, cleaned and optimized in 
several processes before it is made available for retrieval from databases as animation loops. 
And in film, everything can be optimized for as long as it takes before the canned version 
hits the screen. Given these differences, it stands to reason that both in games and especially 
in the performing arts, photorealistic images have not been a top priority. 

The emerging convergence and the increasing use of technology is seen by theorists 
from the fields of film and performing arts as a distinctive feature of digitalization. While 
computer games are digital by definition, film has gone through a groundbreaking transi- 
tion from analog photographic to digital recording. However, theorists have other features 
in mind when they use the term digital cinema. According to Lev Manovich, the constituent 
element is essentially the combination of real image components (conventionally recorded by 
means of a lens) with digitally generated 3D spaces or 3D characters. In his view, the hybrid 
form leads to a particular case of animation film, in which live-action footage is used along- 
side many other image sources (Manovich, 2001; 2016). In addition, these phenomena can 
also be understood as defining components —among many others — of the term post-cinema 
(Denson/Leyda 2016, Iseli et al. 2021). 

An approach similar to that taken in film studies is also used when determining the 
theoretical concepts of digital theater, digital dance or digital performance. These new forms 


of expression are primarliy defined by their hybrid quality, to which digital technology is 


intrinsic (Dixon 2007; Salter 2010, Boucher 2011). In order to distinguish digital theater 
from traditional theater as well as from a broader sense of digital performance, Nadja 
Masura lists the presence of digital technology, both in the creation of a performance and its 
interactivity as a primary characteristic. She also assigns importance to fundamental ingre- 
dients that make up theater in the first place, such as the presence of verbal communication 
or the co-presence of audience and actors. (Masura 2020). Another parallel to film studies 
is that the integration of new digital technologies can also be seen as one of many defining 
components of post-theatre or post-dramatic theatre (Lehman 2006). 


THE ALIVE, THE DEAD AND 
THE UNCANNY 


In film, there is the traditional distinction between /ive-action and animation. The terms 
clarify how moving images are created. Live-action refers to what happens in front of a 
running camera that records multiple frames per seconds in the profilmic situation (Souriau 
1951) and it usually simply refers to shooting 
with actors. Whereas animation in the conven- 
tional sense describes the process to produce 
moving images on the basis of single frames that 
can create the illusion of movement when played 
back at the standard rate of 24 frames per second 
(stop-motion animation). If we look at charac- 
ters we could simply say that either a character 
moves on its own during the shooting process 
(and is thus alive), or a non-living representation 
needs to be set in motion (brought to life) through 
animation techniques.' But from very early on in 
film history the two basic approaches were mixed 
and formed a category that later was defined as 
live-action/animated with real actors appearing 
in cartoon worlds or cartoon charac- 


ters in photorealistic words. An early 


WHO FRAMED ROGER RABBIT? 
ROBERT ZEMECKIS (1988) 


example of a hybrid production were 
the film series Alice Comedies (Walt 
Disney Company, 1923-27), and a 
very successful historical example later 
on was Who Framed Roger Rabbit? (Robert Zemeckis, 1988). This brings us to the basic 


constellation that characters played by living actors meet animated or—to follow the logic 


of the terminology — dead characters, which at the same time appear to be very much alive.’ 

The hybrid image production, in which a real human acts side by side with a synthetic 
non-human, has become quite common in mainstream cinema and it certainly has become 
substantially more popular in the last 20 years. It mainly started in the 1990s— with a few 
exceptions a bit earlier—, which is due to the growth of computer generated imagery CGI 
and 3D-animation in the ever expanding area of visual effects (Prince 2011). 

The main shift towards realism that has taken place since then, regarding synthetic, 
animated characters, occurs on two levels: Firstly, in terms of photorealistic surface textures, 
due to the growing capacities of computer graphics, and secondly in terms of precise human- 
like movements, based on motion and performance capture technology. With this particu- 
lar technology, animation films eventually obtained what we would call a proper shooting 


situation, or in Souriau’s terms, a profilmic reality. 


1 


There are also methods for the 
mechanical or electromechanical 
animation of puppets or artificial 
creatures during a live-action 
recording process. The associated 
term here is animatronics (formed 
from animation and mechatronics, 
a term derived from robotics). 

Cf. Flueckiger 2008. 


N 


The special effects in Roger 
Rabbit were still basically pre- 
digital and were done with optical 
printers, puppetry, and stop- 
motion animation (Wolf 1995). 

In computer animation the frame- 
by-frame technique of analog 
times is replaced by keyframe- 
animation. 
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GEMINI MAN, ANG LEE (2018) 


The growing importance of the live-action/animated genre gave rise to realistic virtual 
characters at a large scale. However, the term virtual character doesn’t necessarily imply a 
human appearance. But it most certainly means— following the long tradition of anima- 
tion film—that CGI characters have human-like movements, human mimicry, human-like 
emotions and human personalities. These anthropomorphic creatures are well-known and 
their naturalistic appearance and behavior have attained a level of perfection. For example, 
Golumn from the Lord of the Rings and The Hobbit franchises (Peter Jackson 2001-2014) or 
Ceasar from War for the Planet of the Apes (Matt Reeves, 2017), both created with the help of 
performance capture by British actor Andy Serkis. 

At the same time, the photorealistic digital human remained the last frontier of 
the VFX industry. The human look-a-likes that were referred to as virtual humans, virtual 
actors, synthespians, digital doubles or avatars first appeared only partially, or in short 
scenes, as early as the 1980s. In the ensuing years they were often kept in the background, 
multiplied to simulate crowds, as in Titanic (James Cameron 1997), or used as digital dou- 
bles to avoid dangerous stunts as in Matrix Reloaded (Lana and Lilly Wachowski 2002). But 
as soon as they appeared in larger portions of the films, left the background and aspired to 
be leading characters with dialogue, close-ups and emotions, they got into trouble. Final 
Fantasy: The Spirits Within (Hironobu Sakaguchi 2001) was the first animation film which 
was entirely based on photo-realistically rendered characters. But the audience acceptance 
was rather weak, as it seemed difficult to emotionally relate to the still quite synthetic 
characters (Butler and Joschko 2009). This is connected to the so-called Uncanny Valley 
effect. First introduced by Masahiro Mori in 1970, the term describes the phenomenon 
that when artificial characters become similar to humans, the empathy curve of an audience 
suddenly drops sharply and only rises again when the characters appear indistinctively close 
to real humans, thus giving the graph the shape of a valley (Mori 1970). 

The debate about the uncanniness of computer-animated characters was conducted 
with vehemence with regard to films such as Robert Zemeckis’ Polar Express (Robert 
Zemeckis, 2004), which relied entirely on performance capture technology (Flueckiger 
2008, Aldred 2006, Kurtz 2004). But later, due to new aesthetic concepts and the techno- 
logical progress demonstrated in, for example, The Curious Case of Benjamin Button (Fincher 
2008), discussions of the uncanny valley effect seemed to recede into the background 
(Flueckiger 2012, Perry 2014). However, when it came to awakening the dead critical views 
reached a new peak and led to ethical concerns. 
In Star Wars Rogue One (Gareth Edwards, 2016), 
British actor Peter Cushing, who died in 1994, 
was digitally reconstructed and joined the cast 
again, appearing alongside a digitally reproduced 
Carrie Fischer. 

In Ang Lee’s Gemini Man (2018), the digi- 
tal twin was finally assigned a leading part. In this 
thriller, the main character meets a much younger 
clone of himself. For this purpose, a state-of-the 
art digital double of Will Smith was produced 
and made significantly younger. In this context, 
the actor behind Golum and Ceasar, Andy Serkis, 
stepped forward in an interview and emphasized, 
that “there are a few serious problems in that we 
can create photorealistic characters, that we can 


digitally rejuvenate actors or digitally retrieve recordings of deceased 


actors” (Pennington 2019). Labeled by the Rolling Stone magazine 
the “king of post-human acting” (Hiatt 2014), Serkis cautioned that 
the technology allows the film studios to do what they want with these digital copies of real 


humans, and the legal issues seem complicated (Pennington 2019). 


PLAYERS, THEIR AVATARS 
AND LOW LATENCY 


In the field of computer games, one thing counts above all else. The graphics that appear 
on the display in front of the players must be calculated in real time. For decades, this basic 
requirement has led to a rather modest visual quality. In the first games in which humans were 
represented, the so-called “sprites” consisted 
of only a few pixels. They were the non-static 
elements within two-dimensional games, 
moving independently of the background 
and were player-controlled, as for instance 
in the arcade game Basketball (1974). The 
player controlling his digital representation 
in the game has been a characteristic feature 
of computer games from the beginning. And 
with the increasing complexity of games, 
it has evolved into what is often described 
as the player-avatar relationship (Gazzard 
2009; Banks/Bowman 2013). It implies a 
role-play related experience that moviegoers 
don’t expect to have. 

With the expanding graphic capacities of computers, the number of pixels per 
character was on the rise and in the Nineties a landmark shift took place towards 3D graphics 
and avatars with three-dimensional bodies. Later, games went online, which limited 
creative animation solutions, due to limited internet bandwidths and the need for the simul- 
taneous representation of actions across a high volume of players. The visual style remained 
mainly reduced and cartoonish, but the enhanced interactivity through the introduction of 
“massively multiplayer online role-playing games” (MMORPG) led to an increased identi- 
fication between gamers and their avatars (Blinka 2008, You et al. 2017). 


BASKETBALL (1974) 


Eventually the characters’ design became more 


came into play as well. Obviously, the standards 
here seemed different than on the big silver 
screen and the uncanny valley never had a com- 
parable significance, as computer games, unlike films, had never been considered a photo- 
graphic medium and thus were hardly connected to corresponding expectations. But now 
that the graphic performance passed a certain threshold, cinema has become an important 
reference, both in terms of visual style and narrative approach. Cinematic Triple A Games 
that employ a third person view, like Heavy Rain (2010), The Last of us (2013), or Detroit: 
Become Human (2019) are considered milestones in the development of high-quality visuals 


and characters. 


HEAVY RAIN (2010), THE LAST OF US (2013), 
complex, and the new frontier of photorealism DETROIT: BECOME HUMAN (2019) 
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THE HERE, THE NOW AND 
THE MEDIATED 


The integration of media content has a long and rich tradition in theater and dance and 


goes back to the 1910s, when film projections began to be included in stage performances. 


In addition to providing alternate backgrounds or providing additional content layers, the 


FORBIDDEN ZONE (2014) 
BY KATIE MITCHELL 


new practices also included the projection of filmed 
characters (Dixon 2007; Salter 2010). With the 
introduction of portable video equipment in the 
1970s, the inclusion of media content increased 
rapidly. The use of video also offered the possibility 
of live transmission, which was frequently used 
as it corresponded to the liveness of the theatrical 
event (Georgi 2014). The fact that the video foot- 
age was captured live in the same time and space, 
was—and is—usually made transparent to the 
audience. Frank Castorf is a prime example of a 
director who repeatedly emphasized medial play 
by employing multiple cameras and including live 
editing (Carlson 2008, Kassay-Schuster 2017). Or 
similarly, Katie Mitchell who uses video to create 
an atmosphere of “live cinema”, as for instance in 
Forbidden Zone (Mitchell 2014) 

The step towards digital characters that are 
animated by means of motion capture technology 


became possible in rudimentary form in the mid- 


Nineties through the use of mostly abstract forms. An example of this period is Biped 


(1999) by Merce Cunningham, in cooperation with the artist duo Shelly Eshkar and Paul 


Kaiser. At that period the motion capture data was often still pre-recorded in order to 


BIPED (1999) BY 


arrive at optimized animation sequences. Other 
groups of that period aspired to achieve real-time 
animation in their performances. Ruth Gibson and 
Bruno Martelli of the Igloo artist group achieved 
double appearances of dancers and their “dual iden- 
tities” in their 2000 performance Viking Shoppers 
(Dixon 2007). 

In order to confirm that the avatar’s perfor- 
mance is indeed being created live and is not the 
result of a pre-recording, the audience usually gets 


to see both the performers (in motion capture suits) 


and their projected avatars. Even though the technical equipment is capable 


MERCE CUNNINGHAM ofa so called rea-time-processing, there is always a slight delay between a 


movement and its processed and projected counterpart. The so-called latency 


has been improved considerably, but still today it’s always there, close to the 


threshold of noticeability, but giving the double appearance an additional artificiality, as the 


avatar seems to inhabit a slightly different time zone. 


On large theater stages and with ambitious visual concepts, the required technical 


effort can reach enormous dimensions. In the Shakespeare anniversary year of 2016, it was 


the character Ariel of The Tempest (Gregory Doran, Royal Shakespeare Company, 2016) 


that could be experienced as a digital avatar animated live by actor Mark Quartley on stage, 


entangled in overwhelming digital stage designs. In addition to a large volume, state of the 


art performance capture system, this required no fewer than 27 projectors and a massive 
array of the latest computer technology from the project partner Intel. (Billington 2016, 
Jimenez 2017). 


DOUBLE TROUBLE 


In his review of the premiere of The Tempest, critic Michael 
Billington (2016) of The Guardian described his experience of 
seeing the Ariel avatar simultaneously on stage with the actor 
Mark Quartley. “It is all impressive, even if it creates the odd 
sense that we are watching a double Ariel” (Ibid). This indeed 
describes a basic dilemma that inevitably arises when working 
with digital avatars, when at the same time the live character 
of the situation is made transparent. Certainly, it would be 
possible to have only the avatars appear on stage, while the 


actors’ movements could be tracked in a separate room. How- 


ever, this would deprive the actors of their 


very purpose for being on stage and prevent 


the necessary co-presence of the audience. It THE TEMPEST (G REGORY DORAN ’ 
would also merely look like a preproduced ROYAL SHAKESPEARE COMPAN Y, 2016) 


video and not like a prestigious, cutting-edge 


digital live production. The simultaneous 

presence of actor and avatar eliminates these problems altogether. The price for this, how- 
ever, is the effect of duplication that critic Michael Paulson of The New York Times referred 
to as the “the double event” (Paulson 2017). 

The descriptions of the double effect caused by digital technology is all the more 
interesting as it seems to echo a more fundamental double phenomenon inherent to perfor- 
mance. In his comprehensive study on the performative arts, Marvin Carlson cites ethno- 
linguist Richard Bauman who noted that performance always involves “a consciousness of 
doubleness, through which the actual execution of an action is placed in mental comparison 
with a potential, an ideal, or a remembered model of that action” (Carlson 2013). And 
sketching the tradition of double concepts within the realm of digital theater and dance, 
Steve Dixon summarizes various phenomena and forms of doubles. (Dixon 2007). Thus, the 
“odd sense of the double” that The Guardian traced back to the spatial co-presence of actor 
and avatar in The Tempest (Billington 2016) echoes a whole variety of double concepts and 
even might have been influenced by the aforementioned temporal double phenomenon due 


to the latency of motion capture systems. 


PRESENCE, ABSENCE AND 
BOTH AT THE SAME TIME 


Presence and Absence is an artistic research project of the Immersive Arts Space at the Zurich 
University of the Arts that explored the phenomenon of double presence with virtual 
characters on stage and worked on playful solutions to overcome the scheme of doubleness 
without losing the transparency of liveness. The initial research interest arose from the earlier 
project TwinLab, a co-production with the experimental theater group Zuni Icosahedron in 
Hong Kong. In both locations the movements of dancers were simultaneously transferred to 


abstract 3D avatars by visual artist Tobias Gremmler and made visible in video projections. 
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PRESENCE AND ABSENCE: THE AVATARS Despite the distance of more than 9000 kilometers, the per- 
APPEAR WHEN THE DANCERS ARE HIDDEN _ formance could be followed iy Ir i both cities. A fast 
BEHIND THE STAGE ELEMENTS. data exchange enabled the virtual interaction of dancers and 


their avatars, simultaneously and at a distance. 

With the Twinlab performance, the research team 
became aware of the dilemma of double presence. Visitors of 
the simultaneous performance described their experience of constantly having to decide 
whether to pay attention to the real actors or the virtual figures. After the debriefing session, 
Tobias Gremmler encouraged the research team to further explore the interaction between 
performers and their avatars in this respect. This resulted in the artistic research project, in 
which Gremmler’s virtual characters from the TwinLab production served as a basis (Iseli 
2021). 

In close cooperation with Gremmler and the team, set designer Mariana Vieira 
Gruenig developed movable stage elements that allowed the dancers Aonghus Hode and 
Lucas del Rio Estevez, who were equipped with motion-capture suits, to disappear behind 
them while their avatars were projected onto the front of the elements. Conversely, the ava- 
tars were to disappear as soon as the dancers became visible. Technically, this is achieved 
with projection mapping, which is the expertise of engineer and artistic practitioner Martin 
Fröhlich, who developed open-source software for handling dynamic projection situations. 
The basis for this is a digital 3D model of the real space in which the projection takes place. 
All projection-relevant elements must be transferred to the virtual model true to scale, 
which is made possible with the dynamically captured position data of a motion capture 
system. This enables the software to precisely map the projected avatars onto the moving 
stage elements. As soon as a digital avatar moves over the edge of a projection surface, the 
projection switches to blank, so that only the real dancer becomes visible. Thus, the alterna- 
tion of presence and absence results in a playful transparency. At the same time, it becomes 
apparent for the audience how the animation of the digital avatars comes about without 
being confronted with a rivalry for attention arising from the permanent double presence. 

An important further feature of the stage elements is that the dancers can step 
through them. This is made possible by a surface that consist of elastic white ribbons strung 


together, instead of a solid screen. Thus, the dancers can push away the elastic white bands 


with their bodies. They can also stick their heads, arms and legs through the flexible surface 
so that they appear only partially visible. And they can also simply jump through the wall of 
ribbons and thus quickly disappear or emerge. 

The state of being both real and virtual at the same time, when the dancers remained 
partially visible, proved particularly fruitful in the exploratory approach. The spectators wit- 
nessed a character that appeared to be half avatar and half human, or that slowly passed 
from one state to the other. Finally, aesthetically, the rule that either only the dancers or 
their avatars should be present proved too rigid. By changing the software, this basic prin- 
ciple could be switched on and off during the performance. Alternating with the playful 
transformations, phases of simultaneous presence were introduced. This combination proved 
to be varied over a longer temporal distance and led to more dynamic sequences overall. 

And what does all this have to do with artistic research? The project Presence and 
Absence consisted of experimental settings that produced explorative sequences with ele- 
ments of the stage set, virtual characters, projection mapping and the dancers’ play. Over 
several iterations, sequences were reset, discarded, adjusted, refined, and further developed. 
At first sight, this process seems indistinguishable from a standard rehearsal process. Typical 
elements of an artistic research scheme are a methodical approach that includes continuous 
reflection, a documentation of the process to ensure comprehensibility for outsiders as well 
as transferability, and an exchange with the peer community by means of publications and 
presentations. Throughout the process, guidance comes in the form of a precise initial ques- 
tion that makes reference to the state of the art. In this particular case it was, “How can the 
simultaneous stage presence of performers and avatars be transformed into a creative inter- 
play of presence and absence but still guarantee a transparency of the process?” 

The output of Presence and Absence is less a completed production but rather a set 
of prototypes. The primary goal is to enter into an exchange with the peer community 
of digital artists and the artistic 
research community by means of the 
publication of the results. Secondly, 
prototypical tools, ranging from 
open-source software to our newly 
acquired practical knowledge can be 
made available to artists in a variety 


of fields for further development. 


TWINLAB: SIMULTANEOUS PERFORMANCE 
BETWEEN ZURICH AND HONG KONG. 
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IN CONVERSATION 
WITH NEIL NEWBON 


ANTON REY 


ZURICH JANUARY 2020 


Anton Rey: Neil Newbon, you have worked in numerous countries all over the world. In 
August 2019, as part of our international conference to the research topic on our SNSF 
project ‘Actor & Avatar”, you trained for the first time with Swiss, German and Austrian 
actors and actresses. Would you say there is a notable difference in approaching, sort of 
locally marked depending on acting traditions? 

Neil Newbon: I think that acting craft has been shaped in Europe both east/west and in 
the Mediterranean for so long that there is a commonality that runs through this part of 
the world. Acting techniques may different between Laban, Method, Misner etc etc but 
ultimately Europe has such a great rich history of writing, theatre, film and acting craft 


that European actors tend to feel similar to one another in these respects and by and large 


NEIL NEWBON 


in their approaches to character. I think there are noticeable differences in approach when 


comparing say, Japanese actors or Indian actors to a German or a British actor, for example, 
due to different cultural influences and storytelling tastes than our particular continent did. 
Of course, it’s worth saying, that at their hearts, all stories are universal and applicable to 
all Humans, even if their respective backgrounds, methodologies and palates differ in some 


respects. 


A.R.: What was the aim of the three days’ workshop and how was it structured? 

N.N.: The aim of the course was to take the student performers abilities and knowledge— 
some from multi disciplines (dance, movement etc) as well as acting crafts, some whom had 
not a great deal of acting or performing beyond the theoretical and assimilate their various 
techniques into the givens of the “Volume”, which is the name of the specific working 
environment of Motion Capture/Performance Capture, which are in turn referred to as 
Mocap/Pcap respectively for abbreviation. 

The major difference between the two historically is that Motion Capture is focused 
on body movements and head only, with or without audio and is usually animation locomo- 
tion (in game movements of player characters and non-player characters) that are intractable 
by a game player, world-building background characters (ie, people at a football stadium or 
in the street), combat and stunts (games, films and actor doubling etc.) or performances by 
say dancers or gymnasts for multi-media use. Performance capture can include all of the 
above but also has face data and finger data and is also with or without audio being recorded. 

As new technical narrative mediums, in the same way that theatre, TV and film are 

technical mediums in which to tell stories, the overall goal is to teach the students how the 
technical systems work, how these constraints can affect their performance, how to start 
thinking in different ways about creating character and implementing movements in the 
animation for use in Film, TV, Games and Interactive theater, how to be able to use this 
new discipline and to show what freedoms and restrictions to their performance the techni- 
cal givens. We encourage the idea that this is a hybrid of theatre and film techniques as the 
virtual camera can be anywhere the actor is essentially “live” throughout the take and we 
train people to adopt a live performance unless they leave the Volume area, as their move- 
ment capture can be used and cannibalised into other moments and the system captures 
everything they do. In this regards the stage is always live and is similar to theatre that there 
is no dead space in the Volume for the actor. 
Alongside my co-founder, Saleta Losada Rodriguez, 
Performance Captured Academy’s workshops are 
designed with a building block approach. We begin 
with an introduction to the technical aspects of the 
Studio using Industry specific jargon as it would be 
on a professional shoot, explaining how it works and 
how this may alter their performance. 

We start in the suits, in the Volume and the 
nature of the workshop is predominately practi- 
cal. We use a step-by-step approach—starting with 
natural basic walk cycles, observing the movements 
of a performer just as themselves, walking from one 
end the volume to the other to allow the performer 
to see character movement based on wants and needs 
and their habitual history to gain a fast insight into 
the need for total body involvement in this type of 


performance and how to start thinking of adjust- 
ments to create new and varied character archetypes. 
This continues with more complex movement-based exercises, adding animation character 
rigs working with different gaits, weight distributions and size changes—we then begin to 


add scene study and creature workshops to take the basic locomotion of Motion Capture to a 
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full acting performance (including vocalization) which we refer to as Performance Capture. 
The student in only 2 or 3 days is exposed to everything that is a-typical in our work in the 
Volume and in our experience, this is enough to get any performer from any discipline ready 


to work on a mocap or pcap shoot applying their craft to the level they possess. 


A.R.: While watching you work, | was reminded of games, but also of the silent movie era. 
The body language is emphasized and stands in the foreground; with the help of avatars 
one becomes aware of bodily expressions which on stage and screen, at least in the educa- 
tion, often stand in the background. During your training phenomena came to foreground 
which we would otherwise encounter e.g. with Commedia dell’Arte or as puppeteers. Do you 


see a direct link to these, silent movie era, CdA or puppeteer? 
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N.N.: Indeed—we often refer to great movement 
artists of the silent film era such as, Buster Keaton or 
Charlie Chaplin. Who created great characters and 
complex sequences with nuance and empathy without 
the need for dialogue or sound. Infact the use of the speech cards in between shots could 
be taken out and the viewer would still understand the story and motivations of the piece. 
Puppeteers often attend our courses and do very well, their understanding of craft- 
ing performances without relying on words to describe the inner thoughts or actions of a 
character does require a certain training and also a certain bravery in trusting themselves to 
be present to create authentic truthful movements whilst not relying on exposition etc. We 
often encourage students to seek out additional training in things like mask work, CdA, 
dance, yoga, method acting, laban, gymnastics for example. Anything that helps quieten 
the mind and get the actors into their bodies is relevant. A good command and control 
of one’s instrument (an acting term referring to the actor as a whole) is applicable to this 
work. Film technique alone will sometimes leave a good actor stilted and wooden physically, 
whereas theater work doesn’t always allow for minute moments that film often focuses on. 
As the cameras are theoretically operating in both a wide and a close-up simultaneously 
(a Schrodinger’s cat effect as you don’t necessarily know where the final camera position will 
be or if that shot will be altered or reused in another place or indeed in another character). 
Thus, these training examples are especially true for Mocap work where often a look or a 
body movement is all that is required to tell the story of the moment. In developing the 
ability to make specific physical choices without generalizing movement quickly when por- 
traying multiple characters during a day’s shooting (which is often the case in these types 
of jobs), the performer will always be able to stay authentic even when portraying minor 


background characters that serve mainly to add to the immersion of the viewer into a world. 


A.R.: How about the face when the body speaks aloud? The Hungarian film critic and 
theorist Béla Baldzs spoke, when describing Asta Nielsen’s acting, of a polyphonic facial 
expression becoming possible when opposing expressions appear in a physiognomy. In 
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a physiognomic chord, as it were, a synthesis of the most diverse feelings, passions and 


thoughts appears—a synthesis that perfectly expresses the multifaceted human being. 


Much to my surprise, you were describing a comparable process when explaining how you 
acted in PLANET OF APES: LAST FRONTIER and described how the Gorilla had a real 


story, a character, you even called it a “personality”? 


N.N.: As a physical performer I don’t see a separation of character no matter the creature 


type, for story purposes most non-humanoid creatures are anthropomorphized to relate 


to the audience—so even if the face is not being captured it is connected, it is still a part 


of the performance as a whole. ‘The character is still rounded and developed in a way to be 


able to play—animals in our own real world have personality and wants and needs—they 


are all alive. 


I work using the idea of an actor’s instrument as a whole, the actor should be focused 


on the character at all times and commit with their intuition and the design of the character 


100% using the totality of their being. Unless there is a tonal/style constraint where the 


face/body cannot be allowed to move for whatever reason (such as the muscularity of the 


creature—as in the case of apes where they must smile in different ways to humans, or 


: men Es i g: 2i 5 


taking into consideration legs 
shape or arm length etc.) which 
is a design constraint the per- 
former will have rehearsed and 
worked into their experience 
before capture, then it should 
be allowed to come in the 
moment naturally. This may 
take practice and experimen- 
tation, but ultimately should 
become habit for the performer 
to work as the character truth- 
fully as you would do in any 
human role. 

Obviously with dialogue 
this further alters this experi- 
ence. I often design characters 
to have habits and quirks that 
I deem appropriate sometimes 
in the face—usually these rep- 
resent to me something about 


the character but not usually 


explained to the audience—but they work with all the facets of character to create a deeper 


well rounded person, a more interesting imaginary being, the aim being to add layers to the 


story and the characters moving through it and immerse the viewer further in the fictitious 


truth of the piece, working with a creature is exactly the same process, they have needs, 


wants, habits and instincts just the same. 


A.R.: Are you consciously training and applying layers of mimic masks on top of each other, 


in order to evoke unsayable expressions which yet hopefully will be “read” by the audience? 


And if so, do you get information from the director or camera operator how close-up the 


picture will be, because this might make a difference on your acting scale? 


N.N.: Constant physical and craft training in new techniques and skills is essential for any 


Actor in any medium—it helps our instrument stay fresh, engaged and interested and allows 


us further range in our work and craft. I often seek out extra training that isn’t directly 


applicable to a project I am working on in the present day but I know and have always found 


at some point will become relevant. 


In terms of the picture, I touched on this above, we are sometimes aware of placement of 
camera, sometimes not. Actors in all mediums work closely with the guidance and voice 
of the director/directing team which may include lead animators as well. We also work in 
Mcap and Pcap, on the premise that each shot is a close up and a master wide at the same 
time—this allows us to be completely free in the Volume and ironically not worry so much 
about this. Marks and specific playing areas in terms of camera angles can come into play 
typically more in scene work and in fixed sequences, but in locomotion it is more often an 
abstract with no set environments necessarily and our own orientation in the Volume in 
relation to not specific camera shot singularly but simply to the floor (or z axis) is actually 
more important—as movements can be mirrored and 
manipulated in isolation by post processing anyway. I try 
to not get too caught up in whether “this is my close up” or 
not and will play each moment to it’s fullest regardless of 


where I am told or believe the virtual camera will be. 


A.R.: Does face tracking make acting artificial, is it the 
glycerine of crying, helped acting, or does it, on the con- 
trary, reveal layers we were not aware of before today’s 
technical possibilities? 

N.N.: I think as the fidelity of the tracking, solving and 
rendering can now bring tremendous detail and nuance to 
a performance in animation, those of us that have been 
doing this work the longest have seen a wonderful tran- 
sition from a heightened level of playing characters with 
exaggerated movements and a more “telegraphing” play- 
ing style leaning towards the theatric, to that of a more 
introspective nuanced filmic style of playing. These days 
we can see an eye twitch, well with a tear, the lips gently 
pucker revealing a subconscious thought etc. whereas that 
may have been lost in less detailed images say 20 years ago. 

Further, I always play scenes with total commit- 
ment and regardless of whether tears, to use your example, 
can be added in afterwards to sell a moment, as indeed it is 
now possible to do so in film. 

I think the best scenes work when you know the 
actor is 100% present in the characters truth with their 
impulses and choices working symbiotically regardless of 
the ability to track and change the face data, everything 
that is possible in film is also possible in Pcap too—you can 
add or take away anything on screen and editing has far 


greater scope as the images can be directly manipulated. 


For Performers we think of it more as “digital make-up” ALL PICTURES FROM THE ZURICH WORKSHOP, ©IPF 


an expression coined by Andy Serkis’, with whom I had the amazing oppor- 
tunity to work with in the Apes franchise in 2015. I agree with him, it is a 
layer of makeup which can be manipulated but the layers of the character still 
come from the actor in their authenticity of the role and anyone watching the 


performance will be able to tell the difference in that respect. 


A.R.: Taiwanese/US American director Ang Lee (EAT DRINK MAN WOMAN, SENSE 
AND SENSIBILITY, CROUCHING TIGER, HIDDEN DRAGON, HULK, BROKEBACK 
MOUNTAIN, LIFE OF Pl), three time Oscar winner and extremely experienced film direc 
tor on diverse genres, also known for his careful, precise and effectful approach to actors, 
described the difference in acting after working on his latest motion picture GEMINI MAN:? 


= 

1 Interview with Tobias Kniebe, 
Stiddeutsche Zeitung Nr. 229, 
4 October 2019 
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“What you feel inside is like a new world for directors. It will not be just the look that 
matters, the whole soul, the whole being will have an effect on you. That’s what I think 
when things get easier with a little bit of digital help. That’s where we going, we are going 
deeper in studying faces, we study people.” Would you agree that new technical possibili- 
ties reveal more than they hide? 

N.N.: Yes, I agree with this—those of us that started in Mcap and Pcap in the early days 
I think all knew this from doing the work, that this was the future, before indeed others in 
the industry gave this style of performance much weight or could see it as another valid art 
form narratively as well as visually. 

In terms of the technical possibilities this evolution offers up, an actor can even 
collaborate with other performers to create an individual role—one artist to perform say 
the voice and face locomotion, one to provide movement in mocap, another one to provide 
stunts and combat, lastly one to perhaps be “skinned” digitally to use as the outside look of 
the character. 

All these artists from different background shooting independently sometimes, can 
now work together for one character, each adding part of their artists soul to it, under the 
guidance of the director and animation team. 

We can ultimately show anything of the character in digital world the only real 
limitation is the story and the imagination of the animators, actors, writers and directors. 
We can study character in the same way that film and theatre can and we can visually add 
and hide anything in the world/of the characters as we desire. Again, it is a choice as to what 
the vision of the piece and story needs. It is of course the same in live action film and in all 
mediums that use animation technology. With technical progress and evolution of fidelity, 


the depth of detail is greater and more manipulable. 


A.R.: Should actors be afraid their image will become a public playground? 

N.N.: This is a lecture in on itself. The answer is—maybe, maybe not. We get paid for our 
usage for each project (voice and/or skin and performance)—contracts are in place to ensure 
fair usage is agreed and unions and bodies are already taking steps to ensure regulation of 
this. Often it is just our movements that are being bought not our actual images, so it kind 
of depends. However there is a good record of companies being transparent about this so I’d 
say for professional actors you will always be told of what they are going to use your work 
and potential your image for and how. Even deceased artists have been licensed for their 
images through their estates and “brought back to life”. Whether this is a good thing or not 
is really a matter for the audience I suppose and as long as there is a correct remuneration/ 
protection of terms in the rights, I don’t see much difference in what actors have been doing 
already for years in tv, film and adverts for example. 

Ultimately when you sign the deal you are making a choice for yourself whether you 
are getting a good deal. Read the contracts, get advice, make up your own mind, know what 
you are agreeing to. 

What is more troubling I feel is people’s images being manipulated in the real world 
outside of entertainment such as pop stars and politicians etc. People that aren't in the public 
eye and can be scanned or photographed and used maliciously. This has and will cause much 
hardship and trouble and is becoming more and more sophisticated by those unscrupulous 


to do so. 


A.R.: Will some day the animator get the Oscar for performing a character which only ever 
existed as digitally mastered puppet, as an avatar? 

N.N.: I have no idea—I think animators currently receive Oscars for the project as a whole 
and as such that I feel would include “puppeteering characters” if it is solely digitally driven. 
However, if you are talking about puppeteering an avatar, which will be captured as a per- 
formance which would be done shot by shot in real-time, then that surely is an acting award 


and could be certainly credited as such. 


FACING 
CHARACTERS. 
A VOLUMETRIC 
INTELLIGENCE 
FRAMEWORK 


VICTOR PARDINHO, PIA TIKKA 


In the enactive virtual reality installation The 
State of Darkness (Tikka et al. 2022) a participant 
is facing a strange prisoner Adam B. in a stressful 
interrogation situation without having any clue 
of what binds them two together. Distant sounds 
from the prison corridors frame their encoun- 
ter inside the prison cell abstracted to darkness. 
The behaviour and facial responses of the virtual 
character are modified by the real-time feedback 
from the participant’s psychophysiological data, 
however, in a way the participant has no control 
of. The experience builds up around the partici- 
pant’s efforts to try to read the facial expressions 
of Adam B. Is he someone to trust, or does his 
restless eye-gaze signal deceitfulness. 
Simulating any stressful human-human FIG.1 


encounter, in the State of Darkness, the STATE OF DARKNESS 


digitally constructed humanlike Adam 


B. behaves as having a mind of its own. 

‘The human-to-human encounters rely on the bodily gestures and verbal utterances of 
the interacting parties, but above all, they are guided by the facial non-verbal expressiveness 
of the two persons. The face is "constituted in as well as constitutive of social interaction" 
between two people (Haugh 2009: 11). In this paper we extrapolate the understanding of 
the dynamics of human face-to-face encounters in everyday world to the encounters between 
a human and an artificial character in virtual narrative context. We draw on Erving Goff- 


man’s seminal focus on the facial micro-movements and ritual expressions as those events 
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that determine the nature of the facial encounter during copresence (Goffman 1967: 1). The 
virtual characters face with its eye-gaze, head position and micro expressions are inter- 
preted as one would interpret a human face in a similar social situation. This despite the fact 
that the artificial humane face is a digital machinery driven by a real-time adaptive intelli- 
gent system (AI), or perhaps by a more rigid, rule-based computer program. 

In this paper the novel notion of volumetric intelligence (Pardinho 2019) refers to 
the technical constitution of a tridimensional machinery of a humanlike artificial face. 
‘The attribute of volumetric means something can be created or measured in terms of three 
dimensional space. Intelligence, in turn, is originally related to human cognition, today also 
related to the term “artificial intelligence” (AT), i.e. computerized intelligence that replicates 
or copies dynamics of human intelligence. In the current connection, the notion of volumet- 
ric intelligence suggests that an humanlike 3D-face can be harnessed with dynamical com- 
puter-generated behaviors so natural that the face may be interpreted to be a face of another 
human. This would be the case in the virtual reality installation the State of Darkness. A 
co-presence is constituted between the human and the artificial character that appears hav- 
ing a mind of its own (Tikka et al. 2020). The term co-presence refers to the sense of “being 
together” in a mediated environment (de Greef/Jjsselsteijn 2000; Ho et al. 1998), involving 
mutual awareness of the other (Heeter 1992). The way the humane character shows concern 
and feelings about what happens to the things and people around it makes us care about it 
(Bates 1994: 122). Similarly, as in real-life, or in movies, also in the encounters in the virtual 
worlds, the mind and mood of the people should reflect the dynamical changes of the world 
and people around them. 

The following discussion is two-fold: First we describe the process of creating an 
artificial character Adam B. for our project The State of Darkness by means of low-cost pho- 
togrammetry techniques, digital software automation, and with a relatively minimal work- 
load. The description will follow the method developed in Pardinho (2019). In the second 
section we describe making the character alive, as well as the design principles behind the 
facial interaction between the participant and the humanlike character Adam B. We con- 


clude with the challenges related to facial inter-acting for and with the artificial character. 


CREATING THE VIRTUAL 
CHARACTER 


Since Digital Emily, the first humanlike avatar of a live actress, was introduced (Alexander 
et al. 2008), a range of methods have been developed for creating humanlike characters, 
including the pipelines presented at the Actor and Avatar conference, ZHdK, by Derek 
Bradley from Walt Disney Research Studio Zürich (https://studios.disneyresearch.com/ 
digital-humans/) and Matthias Wittmann from Digital Domain (https://www.digitaldo- 
main.com/digital-humans/). While characters developed by industrial studios highlight the 
possibilities of the photogrammetry technologies, more time and cost effective methods 
need to be created. ‘The pipeline described here, is intended mainly for the use of individual 
media-artists and low-cost content developers, who need to have tools for fast proto-typing 
of characters for their projects. 

The character creation starts from capturing an actor in a tridimensional space. By 
taking several photographs of the actor from different angles (preferably 360°), it’s possible 
to calculate their position and unique shape in tridimensional space. This photogramme- 
try technique “encompasses methods of image measurement and interpretation in order 
to derive the shape and location of an object from one or more photographs of that object” 
(Luhmann et al. 2007: 2). The content generated with photogrammetry techniques allow 


multiple viewing points to the virtual character in tridimensional environments (e.g. Virtual 


Reality, or Augmented Reality). When implemented in the virtual environment, the par- 
ticipant may walk around the character and even interact with the character in a relatively 
natural manner, if such an option is provided in the design of the experience. 

The workflow for capturing an actor in a photogrammetry booth is quite different 
from capturing an actor’s performance for a traditional movie scene. Instead of a moving 
performance, the actor takes one by one a set of key facial expressions and holds his face 
still while the synchronized multi-camera system simultaneously records each expression 
from multiple angles. A library of photographs of key expressions of a specific actor may be 
also acquired from commercial libraries. For instance, our character for the VR installation 
State of Darkness, depicted in the images of this article, was retrieved from http://www.tri- 
plegangers.com. Once the set of the live actor’s expressions required for the specific project 
are completed, they are further processed by the photogrammetry software, which generates 
3D models from the 2D images using a series of processes powered with computer vision 
algorithms. The four stages of the workflow are briefly discussed below. (For more details 
see Pardinho 2019.) 


CAMERA ALIGNMENT 


The first step in the post-processing workflow is the alignment of the camera position for 
each captured photograph. The software Agisoft searches for common features in each pho- 
tograph and matches them pixel by pixel (Agisoft 2016). Using triangulation algorithms the 
software generates the probable position and orientation of the camera at the moment it took 
the picture (Warne 2015). 


DENSE POINT CLOUD 


After the algorithm has generated a sparse point cloud 
of the actor based on the photographs, it further gen- 
erates what is called a dense point cloud (Fig. 2). This 
included depth maps of each photo. By analyzing the 
position of the photographs and the generated sparse 
point cloud data, the algorithm compares the RGB 
values, meaning the colour information, between the 
tridimensional points. By this comparison, it’s now 
possible to generate a dense point cloud, a much larger 
collection of 3D points in the space representing the 
captured object. Together with the information from 
the last step, knowing the photographs position and 
the sparse point cloud data, it produces a much higher 
amount of small 3D points in the space with RGB 
values created by analyzing the colour data of 

the photos (Koutsoudis et al. 2014). FIG. 2 á 


DENSE POINT CLOUD 


3D MESH GENERATION 


With the dense point cloud generated, now 
containing also RGB values for each 3D point 
in space, a 3D mesh of our captured actor can be generated. At this stage, the software 
implements a Structure-From-Motion (SfM) and dense multi-view stereo-matching algo- 
rithms (Koutsoudis et al. 2014) to create a mesh triangulation. This enables generation of a 
3D mesh, or 3D model, of the scanned actor. 
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3D MESH DECIMATION 


One more crucial step has to be performed before the new 3D asset (Adam B., our digital 
3D actor) can be exported. This process is Mesh Decimation (Kobbelt et al. 1998), which 


consists of decimating the numbers of polygons from a tridimensional mesh. This makes the 


3D READY ASSETS FROM DIFFERENT 
SCANNED FACIAL EXPRESSIONS. 


object lighter and easier to manipulate, requiring also less computer 
processing power. 

With the set of 3D models with different expressions, (Fig. 3) 
it is now possible to move the work totally inside a virtual tridimen- 
sional workspace in a real-time game-engine. When the character is 
inside a real-time engine, it can be prepared to perform facial expres- 
sions by morphing between the different facial expressions. At this 
point it’s possible to connect the facial expression system to other real- 


time workflows, which will be discussed in the following chapters. 


GIVING THE 
CHARACTER 
A MIND 


In this section we give an overview of the technical and 
artistic principles related to the facial interaction between 
the participant and the humanlike character Adam B. of the 


State of Darkness. The discussion is extended to future work 


in the field. The artificially intelligent systems based on Neural Networks and Machine 


Learning applications allow to drive the character’s facial expressions in real time over the 


network in such a way that it seems alive. Furthermore, complex interactions between a 


human and virtual character can be created to the extent that the character generates new 


expressions based on the learned, giving it an appearance of having a mind of its own. 


FIG. 4 
OPEN SOUND CONTROL SYSTEM INSIDE expression, producing an estimate 
UNREAL ENGINE ENVIRONMENT. of values that can be categorized 
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CONTROLLING CHARACTER’S 
FACIAL EXPRESSIONS 


To control the facial expressions of the volumetric 3D 
replica of the actor in the Unreal Engine (UE), we used 
Open Sound Control (OSC) (Schmeder et al. 2009, 
2010) and Unreal Engine’s Blueprint programming 
system (plugin; Buisson 2014) (Fig. 4). The application 
FaceOSC (McDonald 2012) allowed us to match in 
real-time the facial feature data from the viewer’s web- 
cam to the facial feature data of the 3D character in 
UE, modifying them accordingly. 

In this process, we used Machine Learning 
software Wekinator (Fiebrink/Cook 2010) to train a 
range of 3D character’s facial expressions based on the 
recorded data of human dynamical expressions. In case 
of neutral facial expressions, for example, Wekinator 

will gather samples of data values 


that represent a human neutral 


as a neutral expression. Similarly, 


recording smiling facial expressions, the accumulated sample data will be categorized to 
constitute this particular facial expression. By this training method, the application is capa- 
ble of learning which range of data represents a neutral, smiling, and in principle, any other 
specific expression. Eventually, the virtual character’s facial behaviour can be controlled 
according to the trained data from Wekinator. The application performs the character's 
facial expressions in real-time based on any actor’s facial data recorded by the computer’s 


webcam. 


DESIGNING THE FACIAL INTERACTION 


How the participant experiences the mood and mind of the virtual character is not only 
dependent on the facial or bodily expressions but also of the context within which the 
encounter takes place. In the VR installation State of Darkness, the participant’s face-to-face 
meeting with Adam B. is contextualized within the frames of (i) the participant’s pris- 
on-related imagery from lived experiences, media, news, books, 
films, and so on, and, (ii) Adam B.’s behavior. However, these 
two domains cannot be separated in the real-time recording of 
the participant’s physiological responses, thus, in the following 
the recordings by the biodata are assumed to be influenced by 
both (i, ii). While the context-awareness cannot be emphasized 
enough, in this section, however, we focus on the creation of the 
facial expressions of an artificial character. The facial expressive- 
ness of Adam B. includes two different layers: (iii) the pre-re- 
corded default behavior based on the narrative context (without 
the participant’s feedback) and (iv) the real-time enactive behav- 
ior (adaptive to the participant’s feedback). (Fig. 5) 
For the State of Darkness, the default facial FIG.5 


behavior (iii) was acted by a professional stage per- PARTICIPANTS EXPERIENCING 
former, who listened to the full soundtrack of the THE STATE OF DARKNESS. 


VR experience while simulating with facial expres- 
sions the imagined dynamical changes in Adam B’s 
mindset. This professional acting-out the default states of the inner mind of Adam B. is 
considered crucial in the process. The facial motion capture software Face cap in iPhone 
(© Bannaflak) allowed to export recordings to FBX with mesh, blendshapes and animation 
data. The motion capture data was then implemented to the timeline of the facial feature 
data of the character in UE. The feedback from the psychophysiological recordings (iv) was 
conducted using commercially available biosensors. During the VR experience, the excite- 
ment and arousal of the participant triggered a direct response from Adam B. ‘The stronger 


the emotion the participant was feeling, the stronger the emotional response of Adam B. 


MEASURING PHYSIOLOGICAL SIGNALS 


For the sake of installation practicalities in an open public space, we used the Empatica 
4 wrist sensors, which provided relatively accurate measurements of the changes in heart 
rate (HR) and electrodermal activity (EDA). HR and EDA changes provide physiological 
quanta of arousal, aversion, and stress response. The EDA, or skin conductance, is a well- 
known indicator of arousal but is a very relative measure: levels of skin conductance vary 
between people and even same people can have different baseline level and range of activity 
between different recording times due to reasons such as how well the skin was cleaned and 
what is the temperature in the room. Therefore we needed to build a system that dynami- 
cally calibrates to the current level and range of each user. We did this by keeping a running 
window of recent skin conductance activity from which we constantly recalibrated the sys- 
tem to provide accurate feedback of the current changes in the user’s level of arousal. Once 
the calibrated arousal levels were calculated, they were packaged into OSC (open sound 


control) protocol and sent over UDP network connection to the Unreal Engine. 
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CHALLENGES OF INTERPRETING FACIAL EXPRESSIONS 


Considering the complexity of the setting, we assume that the same facial expression in two 
different moments during the interaction can be interpreted in very different ways, imagine 
flashing a smile when hearing a painful cry, or when responding to another person’s smile. It 
thus follows that relying merely to simple basic emotion categorizations based on the facial 
features (e.g. FACS, Ekman and Rosenberg 1997) can be to some extent misleading, when 
interpreting the situatedness of the participant and/or the character Adam B. Thus, we pre- 
fer not to name the emotional states based on pre-categorizations but, instead, use references 
in terms of the temporally unfolding narrative, both as consequences of the past events and 
anticipations of the future events. ‘The facial expressions generated by the unreal engine are 
constellations of multiple feature vectors (numerical data) each assigned with control over 
particular points in the character’ face map and modifiable in real-time. Thus, the momen- 
tary expressions of Adam B. merge as a combination of the time locked default facial data 
(iii) and its dynamical modifications depending on the psychophysiological feedback data 
from the participant (iv). In the installation, the physiological data (iii) also affected the 


context, creating specific changes in the audio-visual prison environment. 


CONCLUSION 


The natural human face as the goal in creation of artificial characters, the live actors’ profes- 
sional expertise remains on the spotlight in the creative processes of each individual media 
work. An actor gives a character its unique personality, the individual ways the character 
expresses attention, intentions, motivation, care, love, and hate. Novel technologies, such 
as 360 degree volumetric video capture of actors in action, will give the actors even more 
freedom to act their characters in volumetric spaces, a topic of our future research. What- 
ever the chosen method, the challenges of creating and controlling humanlike characters 
expand from the technological to the psychological domains, far beyond the challenges of 
creating other animated entities, such as angry birds or speaking cars. This is due to the fact 
that humans are evolutionarily equipped to “read” friendliness or hostility in milliseconds 
from other people’s faces or bodily movements. Such emotive-cognitive abilities are directly 
transferred to the performative settings, be those on the live stage or in audiovisual media, 
where actors enact human behaviors in different narrative contexts. The emphasis of con- 
text-aware reading and interpretation of the facial expressions multiply exponentially the 
complexity of executing control over the adaptive artificial character’s behavior. The chal- 
lenge is best handled by transferring the control of the facial behavior of artificial characters 
to the artificially intelligent systems. As this is an emerging and fast advancing field, we 
envision that the next edition of the State of Darkness may already introduce a fully indepen- 
dent Adam B., with a repertoire of emergent self-generated humane responses to challenge 


the participant facing him on the other side of the table. 
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A NEW GEOMETRY 
OF THE EYE. 

THE CURIOUS CASE 
OF BRAD PITT 


JORG STERNAGEL 


What is the curious case of Brad Pitt? Does the curiosity come from watching and having 
watched the motion picture The Curious Case of Benjamin Button, directed by David Fincher, 
and starring Brad Pitt in the leading male role, back in 2008? What is the curious case of 
both the character Button and its actor Pitt? Might it be about face, or, to be more precise, 
about vicissitudes of the face, changes, processes, developments that are in the face, that are 
rendered visible on the face? 

To begin, let us consider a review of the film, a review by Al Hoff, critic for the 
Pittsburgh City paper, who elaborates on his experience with the film, with the faces, with 
the vicissitudes of the face, with the curious case of Brad Pitt: “At the screening I attended, 
I could hear the audience snuffling, but I found David Fincher’s overly long, glossy account 
of one man’s odd life to be more emotionally distancing than engaging. Liberally adapted 
from F. Scott Fitzgerald’s short story by Eric Roth (whose Forrest Gump this film reminds 
me of), the film tells the instructive tale of a baby named Benjamin Button, born old but 
who, as he grows up, becomes younger. Button is mostly played by Brad Pitt through the 
magic of make-up and digitally appending his head to tiny wrinkled bodies. In a series of 
episodes—intercut with an awkward dying-woman-recollects framing device —we follow 
Button from his childhood in a New Orleans old-folks home (how con- 
venient), through World War II adventures and his mid-life romance 
with a ballet dancer (Cate Blanchett). While entertaining as a large-scale 
fable, Button’s story left many quirky details (a backward-running clock, 
a pygmy) unexplained, while blithely adding contrivances (Hurricane 
Katrina; a useful inheritance). Pitt is game (though you could feel the 
audience relax when he achieved his mid-life perfect Brad Pitt-ness, 
sunburnished and lounging in vintage khakis), but Blanchett seemed an 
awkward fit throughout. Buzton presents a mildly interesting idea juiced 
with a lot of greeting-card sentiment and golden light. Its point doesn’t 
seem especially illuminating: We all age (or unage), and that process is 
rife with regrets and loss. But living life in reverse doesn’t change the 
basic mechanics of the human condition, just as a clock that runs back- 
ward doesn’t change the fact the time still moves forward.” (Hoff 2009) 

In the light of Fincher’s film, what follows, let us consider three 


issues of Hoff’s review, beginning at its end and ending up at its begin- 


ning: those will be discussions of, first, the Brad Pitt-ness he claims, 
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second, the magic of make-up and digital appendage he addresses, and, third, his own 
experience with the film that he describes as emotionally distanced. These three issues will 
subsequently be contextualized by brief paragraphs from F. Scott Fitzgerald’s 1922 short 
story of the same name on which the film is loosely based and will be discussed within 
three paradigms of visibility, a model partly suggested by the sociologist Andrea Mubi 
Brighenti in his essay “Artveillance: At the Crossroads of Art and Surveillance” (2010). The 
first paradigm, dealing with the Brad Pitt-ness, will shortly be presented as the visibility 
of the spectacle, the second, on the make-up and the digital effects, also quite shortly as the 
visibility of recognition, and the third, starting with the reviewer’s experience, more at length, 


as the visibility of being. 


I. THE VISIBILITY OF 
THE SPECTACLE 


Benjamin, once he left the hospital, took life as he found it. Several small boys were 
brought to see him, and he spent a stiff-jointed afternoon trying to work up an inter- 
est in tops and marbles—he even managed, quite accidentally, to break a kitchen 
window with a stone from a sling shot, a feat which secretly delighted his father. 
Thereafter Benjamin contrived to break something every day, but he did these things 
only because they were expected of him, and because he was by nature obliging. 
(Fitzgerald 2003: 67) 


Before the film, Brad Pitt, the actor, as we continue with the help of Fitzgerald, works up 
interests in the work, in his fellow-actors, in props and set designs, in the camera, to bring, 
at least in part, Benjamin Button to life. In sum, his interest, at least on an artistic level, 
would be to create a role. Before the film, we, in turn, work up an interest for 
Brad Pitt, as we have experienced him in numerous films before, as in Seven 
(David Fincher, 1995), Seven Years in Tibet (Jean-Jacques Annaud, 1997), 
and Meet Joe Black (Martin Brest, 1998) and therefore decide to be at the 
movies with him, to watch him in another film. In sum, our interest, on an 
experiential level, would be to meet again. During the film, Brad Pitt offers 
variations of his art, variations we are partly familiar with like the tone of his 
voice, his prononciation of words, his gestures, his posture, his obliging 
‘perfect Brad Pitt-ness’. Let us remember Al Hoff writing: “Pitt is game 
(though you could feel the audience relax when he achieved his mid-life 
perfect Brad Pitt-ness, sunburnished and lounging in vintage khakis).” 
Here, the focus is on the situation of perception during the film, a situation 
that is also influenced by experiences before or after the film. This is where a 
visibility of the spectacle develops, pointing successively lesser and lesser to Brad Pitt’s 
familiar performing choices and my experience of them, but more and more to his star or 
celebrity functions. 

Such a paradigm could be further scrutinized with Richard Dyer, for example, and 
his thoughts about the Heavenly Bodies, where the economic importance comes into play, 
and the star is also presented as being fashioned “out of the raw material of the person”, 
where “make-up, coiffure, clothing, dieting and body-building can all make more or less 
of the body features they start with.” (1986, 5) Or, as Brighenti points out, it could be 
interpreted with Guy Debord and his approach in the Society of the Spectacle, where we are 
separated from the celebrity or star in everyday life as the images we see with him, from 
him, are detached from life and only propose an illusionary form of life’s unity, where Brad 


Pitt is “the spectacular representation of a living human being”, embodying this banality 


by embodying the image of a possible role.” (1962: § 60) At this point, to move to the next 
paradigm, let us further reflect upon why the audience of Hoff’s screening relaxed as soon as 
Brad Pitt achieved mid-life in his role in the film. 


ll. THE VISIBILITY OF 
RECOGNITION 


But one day a few weeks after his twelfth birthday, while looking in the mirror, 
Benjamin made, or thought he made, an astonishing discovery. Did his eyes deceive 
him, or had his hair turned in the dozen years of his life from white to iron-gray 
under its concealing dye? Was the network of wrinkles on his face becoming less 
pronounced? Was his skin healthier and firmer, with even a touch of ruddy winter 
color? He could not tell. He knew that he no longer stooped, and that his physical 
condition had improved since the early days of his life. ‘Can it be-?’ he thought to 
himself, or, rather, scarcely dared to think. (Fitzgerald 2003: 68) 


Can it be, to continue with the film’s very loose adaptation of Fitzgerald’s prose, that the focus 
on Pitt and especially his face relies on patterns of recognition? Certainly, also this film in his 
making, to follow James Naremore in his study Acting in the Cinema, is dependent on “a form 
of communication whereby meanings are acted out” where, once it is released in cinema, my 
“experience of watching them involves not only a pleasure in storytelling but also a delight in 
bodies and expressive movement, an enjoyment of familiar performing skills, and an interest 
in players as ‘real persons’.” (1988, 2) Here, the paradigm of the spectacle shifts to the para- 
digm of recognition as “the interest in players as ‘real persons” remains influential, while the 
“delight in bodies and expressive movement” become central. Here, the film’s decision is to 
offer a basis for orientation from which a “delight in bodies and expressive movement” can 
develop. This basis, this center is the face of Brad Pitt, with all its features, its eyes, ears, nose, 
mouth, hair, even its “network of wrinkles”. Throughout the film, Pitt occassionally appears as 
himself with his own acting body and enters with his own face and his well-known expres- 
sions, but, more often than not, his body in general and his face in particular solely function as 
sources of inspiration and contact: Pitt’s face is partly attached to other actors’ bodies, acting 
out either familiar or different meanings, while being always ready for the close-up, while 
being successively experienced as aging backwards. “Button is mostly played by Brad Pitt 
through the magic of make-up and digitally appending his head to tiny wrinkled bodies”. 
(Hoff 2009) Pitt’s moving, constantly altered face frequently appears at the center of attention 
and offers both an orientation and a dis-orientation, along with an ever-changing body that 
does not always belong to Pitt himself. Here too, “something sharper than a mask is looming”, 
“something sharper” that Roland Barthes sees in the face of actress Greta Garbo, »a kind of 
voluntary and therefore human relation between the curve of the nostrils and the arch of the 
eyebrows; a rare, individual function relating two regions of the face.” (2004: 721) However, 
in Pitt’s face, this “human relation” is re-considered and re-structured in digital codes, inviting 
us to reflect upon a very slippery realm of film experience, where unresolved affects might 
prevail, partly created by, and this is where the first part of the essay’s title comes from, “a new 
geometry” of the face, its rendering, especially in detail of the eye, of its “eye socket structure”, 
as one of the visual artists of the motion picture, Edson Williams, points out, for “the effect of 
age re-juvenation”. The techniques in question are correspondingly named in a meaningful 
way: “Emotion” and “Facts”, facial coding systems, enabling animators to use micro-expres- 
sions to manipulate individual movements such as a dimple or eyebrow twitch, taken from 
Pitt’s own face, aiming to keep Pitt’s/Button’s expressions true to the performances through 


the ages and on to the bodies, in order to keep the possibility of recognition for the spectators. 
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ll. THE VISIBILITY OF BEING 


Up in his room he saw his reflection in the familiar mirror—he went closer and 
examined his own face with anxiety, comparing it after a moment with a photo- 
graph of himself in uniform taken just before the war. ‘Good Lord!’ he said aloud. 
The process was continuing. There was no doubt of it—he looked now like a man of 
thirty. Instead of being delighted, he was uneasy—he was growing younger. He had 
hitherto hoped that once he reached a bodily age equivalent to his age in years, the 
grotesque phenomenon which had marked his birth would cease to function. He 
shuddered. His destiny seemed to him awful, incredible. (Fitzgerald 2003: 76) 


Indeed, there is a process of continuing, of establishing the phenomenon of aging back- 
wards, prominently through and with the face of Pitt, with an attention to detail carrying 
affects that initiate experiences of belonging and non-belonging in human and non-human 
interrelation: But, here, too, “instead of being delighted”, as Fitzgerald proposes considering 
his protagonist looking in the mirror, seeing himself getting younger, for me, too, there is 
an uneasiness, as is for Hoff: “I found David Fincher’s overly long, glossy account of one 
man’s odd life to be more emotionally distancing than engaging.” (2009) The grotesque 
phenomenon of aging backwards that Fitzgerald writes about in his short story is sought to 
bring to the screen and there, too, it shows, even if at least in part in an unforeseen way: the 
bodily uneasiness with and the emotional distance from what is rendered visible on screen 
are centered on and around Brad Pitt’s fragmentized body, following his remaining bodily 
traces in a very slippery realm of film experience, where unresolved affects might prevail, as 
especially the partly animated face of Pitt develops to be ungripping, causing uncertainties 
of how to act within the visual, auditory, and tactile field between the screen, Brad Pitt, his 
digital fragments, Al Hoff, and me, for example. Here, the coexistence with the film actor 
is challenged, and the bodily link to the actor is loosened. ‘The form which is common to 
both the actor’s and the spectator’s visual and tactile perceptions occassionally dissolves, the 
form that both actor and spectator possess, where, according to Maurice Merleau-Ponty, 
“all happens as if the intuitions and motor performances of the other are founded in a kind 
of internal encroachment, as if my body and the other’s form a system.” (2010: 452) The 
form is my body, and what I learn to consider as the other’s body is a possibility of move- 
ments for me; thus, Merleau-Ponty: “We can say that the actor’s art is only a deepening 
of an art that we all possess.” (Ibid.: 453) This possibility is partly vanishing in the face of 
Pitt and his others, on the various bodies, despite the attention to detail and the use of his 
own voice. Ihe visibility of being turns to non-being, and provokes concluding remarks on 
a phenomenology of the face: Details of the face are preceded by physiognomies in which 
moments of facial expressions, gazes, and liveliness abruptly emerge. These moments are 
not registered as empirical data to assemble, or to compose a face, moreover, these moments 
are seen as primarily forces and affects, and we see less a geometrical form than a pure and 
moving expression. While the making of Benjamin Button with the help of the Brad Pitt- 
ness certainly adds to this human experience of the face, it also certainly lacks an important 
ingredient in its rendering: a somehow pure and moving expression, leading to a slippery 
realm of film experience. 

“Through the noons and nights he breathed and over him there were soft mumblings 
and murmurings that he scarcely heard, and faintly differentiated smells, and light and 
darkness. Then it was all dark, and his white crib and the dim faces that moved above him, 
and the warm sweet aroma of the milk, faded out altogether from his mind.” (Fitzgerald 
2003: 83) 

At the end of the film, the dim, indistinct, unclear faces of Brad Pitt fade out 
altogether from my mind and body, and, unfortunately, not even “the warm sweet aroma of 


milk” remains and I am still kept at a safe distance. 


ON ALI MOIN!S 
AVATAR HACKING’ 


ALEXANDER GERNER 


“The project is a platform where through a 100% mechanical pulley system, ahuman 
size marionette and me/performer have the possibility to interact. The machine 
works as a transmitter/translator of movement, with which I will work on demands, 
agreements, affection, intersections, differences, transportation, hybridization, 


accompaniment, anger, violence, manipulation... between me and a double of me.” 
Ali Moini, FIMFA (2018) 


IMAGE 1. 

VIDEO STILL: ALI 
MOINI “MAN ANAM 
KE ROSTAM BOVAD 
PAHLAVAN” 


LISBON FESTIVAL OF MARIO- 
NETTES AND ANIMATED FORMS 
FIMFA LX18, MAY 15-16, 2018, 
TEATRO MARIA MATOS (HERE 
MAY 15, 2018). 
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in Moini’s performance entitled “Man anam ke Rostam bovad pahlavan.” The mysterious Position: 2404. 


into machine — cyberization—and machine into human — anthropomorphism— is reflected 


139 


and strange Persian title of the performance is derived from an Iranian/Persian proverb, of 
one of its most important cultural figures— Rostam—a generous, genius, powerful charac- 
ter of the Persian pre-Islamic myth of the Epic of Kings. In this Persian myth—the only 
one that has been entirely transmitted, fixed in writing by Ferdowsi in the 10th century as 
“Shah-Nameh” — Rostam kills hundreds of giants and demons. Whoever utters the proverb 
proposes a manipulative rhetorical act related to a—I would not say false, but— plagiary 
inheritance of equality of fame and power with Rostam. A possible translation that Moini 
proposes would be: “I am the one, though/since Rostam is and will be the ultimate cham- 
pion” (Moini, December 7, 2020, personal communication). 

According to Moini, the utterer of the proverb hopes that the listener will not find 
out about the cheat because of the confusion created. Thus, in the words of Ali Moini, “the 
proverb is simply a bluff, but as it’s playing with misunderstanding and misleading through 
linguistic games, it’s not an obvious bluff!” (Moini, December 7, 2020, personal communi- 
cation). 

Let us go beyond the language game of cheating and manipulation and ask of Moi- 
ni’s performance: What if the unreal and manipulation are taking over our reality, including 
not only words and actions but the entire physical and fully embodied world, including its 


actions, actors’ roles, and personae? 


2. BEYOND AVATAR DREAMS 
AND AVATARISM 


Moini focuses on the re-materialization of direct body-force manipulation in which the 
approaches of machine readability and digital motion capture tools (Delbridge 2015; Kar- 
reman 2017) are called into question. Moini does not adhere to chimera-identity effects of 


anticipated imaginations of programmed movements within the cultural avatar dream (Fox 
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IMAGE 2. 
VALENTIN BAZAREVSKY AND IVAN 
GRISHCHENKO, RESEARCH ENGINEERS, 
GOOGLE RESEARCH (AUGUST 13, 2020), 
ON-DEVICE, REALTIME BODY POSE TRACKING 
WITH MEDIAPIPE BLAZEPOSE 


GOOGLE Al BLOG: RETRIEVED ONLINE SCREENSHOT FROM 
HTTPS://Al.GOOGLEBLOG.COM/2020/08/0N-DEVICE-REAL-TIME- 
BODY-POSE-TRACKING.HTML 


IMAGE 3. 
MOINI AND HIS AVATAR PLAYING 
WITH THE T-POSE. 


PHOTO TAKEN FROM “MAN ANAM KE ROSTAM 
BOVAD PAHLAVAN,” PHOTO: YANN GIBERT, 2018 


Harrell & Lim 2017). Moini’s performance critically and forcefully reflects his bodily resis- 
tance to future artistic visions such as virtual identity surrogates or avatar dreams. Moini 
does not follow motion-capture performance aesthetics; instead, he critically plays with 
the zero point or pole position of such endeavors, such as in the T-pose, the default shape 
of the humanoid game models used to calibrate and initiate motion capture and render- 
ing. The T-pose is essential as a placeholder for animations (or imaginations) not yet com- 
pleted. However, in a repeated sequence at the beginning, the performer slips out of these 
motion-capture poses—and so does his double on stage after a short delay—as if incapable 
of being captured, as a glitch in a material system of motion and movement appropriation. 
The aesthetics of the avatar pushes us into a post-digital age in which we can track move- 
ments and create movement skeletons, such as with the digital movement program Kinect, 
with body or face-tracking applications, and then apply an “avateering” software such as 
Vitruvius that enables control of any humanoid 3D model, in which not only is motion cap- 
tured in dance and gesture research, but computer-aided anthropomorphic agents become 
part of performances. 

Nowadays we enter fields of high-tech robotics: dance is co-opted as a slave move- 
ment to transpose a movement pattern from one performer to many others, scrapping 
human uniqueness for the posthuman “singularity” (Kurzweil 2005). In a recent research 
paper titled “Everybody dance now,” Chan et al. (2019) put forward the idea that we can 
usurp the dance movement of another dancer/mover, called the source, by transferring body 
movements via algorithms to video image footage of someone else, the sarget, as an elaborate 
uncanny ghosting movement in AI puppeteering. The principle, in this case, is a real-time 
target matching and a “do as I do” motion retargeting that makes “everybody” (including 
several virtually re-embodied targets) “dance” in the same way as the source movement. This 
mechanism should be critically considered a human-mimetic artificially-embodied AI in 
which dance movement is recorded from one person and transferred in a usurpation of the 
other’s strength, a non-synchronized feedback loop reapplied to a targeted dancer. 

This concept of source/target doppelgingers, the limited imitative sameness of mir- 
ror-image avatars or digitally and mechanically reproduced puppets (Wolfson 2018), is in 
steep contrast to Moini’s live performance, which no longer fits into digital doubling motion 
capture tools of digital avatar corpses that have to be animated by the dance moves of others. 
The uncanny ludic play between “death” and “life” is controlled by the equilibrium between 
tension and counter-tension. 

The game is to play with the other’s counterweight resistance and the limits in its 
choreographies of movement to avoid crashing in the danger zone between freedom and 
necessary constraints (Serres 1995: 106-107). Only in the end is the tension of the steel 
cables released, and with rapid moments, the steel cables spring into the air, and the avatar 
form collapses into disorder. If the avatar performance system of Moini is a defense of lib- 
eral avatarism (Fodor 2021), freedom of form and the transmitted, animated, or expressed 
self-image are called into question as the avatar is finally suspended in the air, unbound, and 
the performer freed not from becoming an avatar, but from his double. 

An avatar without a head or eye gaze, such as Moini’s or the chimeras of actors/masks 
without gaze, renders visible the absence of a living human body, in which a human visage 
(Artaud 1947)/human voice is missing. This absence of the human echoes as well in the 
work of the contemporary artist and theater-maker Susanne Kennedy and her posthuman 
avatars on stage: “I see a theatre where the voice no longer belongs to the body, the face is no 
longer the conveyor of emotion (...). In the end, the actor becomes imperceptible” (Susanne 
Kennedy cited in Trueman 2018). Moreover, Susanne Kennedy declares, “I see a theatre 
where the protagonist no longer exists and the stage where he used to stand in the center 
of it all, is filled with other beings— human and non-human. They speak with voices and 
faces that are not their own. They communicate in languages we have yet to learn” (Kennedy 
2015). Moini’s performance resonates with Susanne Kennedy’s vision of a post-digital ava- 


tar theater, although the two take contrasting approaches to its aesthetic realization. 
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In the fields of performative arts such as dance and theater, as well as in cinema, the avatar 
becomes different to a mere playable gaming character. Thus, we can ask what the /ived bodies 
in time factor tells us about the differences between theater, cinema, and dance and the missing 
avatar gaze in Moini: Is it true that cinematic “events” in the sense of transitionless transi- 
tions— or theater, and I include dance— as “reactions to events” (Eisenstein 1949: 6), by which 
the possibility of the other and its gravitas of a persona on stage or screen are given in the 
first place, are rendered impossible or diminished by the introduction of repetitive avatar play? 

Florian Malzacher’s vision of contemporary political theater, Gesellschaftsspiele 
(2020), shows how avatars as digital and perfect representations are re-embodied on stage 
imperfectly and gain human actors’ flaws. Two examples he gives are the android double of 
the actor Thomas Melle in Uncanny Valley (2018) and Susanne Kennedy’s Coming Society 
(2019), as well as other works by the same artist, in which actors play with avatar-masks of 
their own faces, thus emphasizing the singularity of the event of acting live on stage and 
the importance of live characters being unable to repeat hand movements, swallow, tremble, 


breathe, and sweat despite its repetitive Avatar game: 
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IMAGE 4. 
THE SCULPTURE AND SCENIC 
INSTALLATION OF MANNEQUIN- 
CORPSES ON STAGE AS AVATARS 
AND SYMBOLS OF YOUNGER 
SELVES OF THE ACTORS IN 
KANTOR’S “UMARŁA KLASA” 
THEATER OF THE DEAD. 


PHOTO: ADRIAN GRYCUK - EIGENES WERK, CC 
BY-SA 3.0 PL, HTTPS://COMMONS.WIKIMEDIA. 
ORG/W/INDEX.PHP?CURID=24274872 


IMAGE 5. 

VIDEO STILL SCREENSHOT 

FROM THE FILM ADAPTATION: 
TADEUSZ KANTOR “UMARŁA KLASA” 
(DEAD CLASS) 


YOUTUBE HTTPS://WWW.YOUTUBE.COM/ 
WATCH?V=A235HHGFIPS (05.05.2021) 


The perspective of theater is not one from the future, but one 
directed toward the future. A world in which humanity has 
dissolved into nature, technology, or data, theater cannot repre- 
sent, cannot depict. But it can represent the rapid changes from 


a human perspective and explore scope for action. 


“Susanne Kennedy stages a similar game under reversed conditions in Coming 


Society (2019), when she has avatars—inadequately—represented by humans. Their 


trembling, their rapid breathing, their sweat are the connecting channel to the 
audience. Just as little as the Melle-bot can dance, so little can Kennedy’s performers 


embody themselves in a way that would be appropriate for virtual figures.” 
(Malzacher 2019: 46, my translation) 


With avatars that function as playable characters we are not so much concentrated on their 
gaze, but more on what we can do with them as an action play and how we influence the 
avatars movement as a complete character. Humans have gravitas when they mimic and 
play: What does it mean to be human? What is this ritual of doing and enacting theater 
to mimic and play a character in film? It is about a live ritual for the still alive but coming 
dead, about the process of dying— or about playing with the “dead class” puppets or avatars 
of younger pupil selves as a preparation for our loneliness of memorizing the past and our 
self-doubles over time, such as in Tadeusz Kantor’s The Dead Class (1975). The death mask 
and photography have in common that they are both relics of the past of a person or a 


moment of time gone, but preserved to see and touch. 


3. THE ART OF HACKING 
THE DIGITAL BY PHYSICAL 
AVATAR HACKING 


We can describe the challenging movement game of Ali Moini’s performance, which 
includes the realization of a metal-rod avatar as a reflexive and reactive double of the per- 
former on stage, as a play between vectors of force and embodied rehearsals of systematic 
resistance to scalar cultural games of mere synthetic simulation and recursivity of actions 
(e.g., digitized kinetic avatars). The mechanic details that Moini hacks into show strong 
resistance to pressure and require a “veczorialist” position of attention on the part of both the 


performer on stage and the audience that partakes. In the words of Yves Citton: 


“(...) we should not lose sight of the fact that attention itself is 
a vector, pressure, orientation, “direction of effort”, Zuwendung, 
(...) The vectorialist class is not exploitative because of its ‘power 
to move anything and everything”, but because of its requirement 
that “value be realized” in countable terms. Such is THE TRUE 
CHALLENGE OF DIGITAL CULTURES now emerging: how can you 
take advantage of the vectorial power of the digital with- 
out allowing yourself to be imprisoned in the scalar cages of 
digitization? Only the art of interference, the elusive strength of 
hackers, can rise to such a challenge—which is at the heart of the 
attention ecology in the age of its electrification.” (Citton 2017: 78) 


Moini does not define himself as a mere dancer nor as a hacker. I would 
nevertheless call him a deep cultural hacker—taking advantage of 
vectorial and material-embodied aesthetic thinking and mechanical 
movement, power, and strength while interfering in existing patterns of 
artistic gestures to rebuild systems of acting, involve- 


ment, and play within encounters of alterity to install 


proper performative degrees and qualities of freedom. IMAGE 6. 


In this description, Moini’s avatar double performance 
shows movements of resisting and gestures of avatar 
hacking. By contrasting constellations between the 
mediating body gestures of the performer and the 
effected action in the steel puppet we can speak of 
double movement gestures, which have the peculiarity 
of showing that they show themselves (cf. Mersch 2014) 


ALI MOINI “MAN ANAM KE ROSTAM BOVAD 
PAHLAVAN” MOINI HACKING INTO HIS AVATAR 
MOVEMENT PUPPET PHOTO: YANN GIBERT, 2018 
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IMAGE 7. 
ALI MOINI AND HIS AVATAR SLIDE TO THE for the EEE apay Ga Pustiati ey of 
SURFACE OF THE STAGE. PHOTO TAKEN FROM human life’s material-biological necessities. His choreo- 


“MAN ANAM KE ROSTAM BOVAD PAHLAVAN.” 


PHOTO: YANN GIBERT, 2018 


in this doubling effect in the sense of a direct reflection on stage: “a double perception 
because it also makes something perceptible, just as it exhibits the perception of this percep- 
tion, the showing of self-showing, as it were” (Mersch 2019: 219). 

Moini does not see the human body as what lies at the core of his choreographic 
activities, but in his performances he places the human body in a larger context, such as in 
his architectonic mechanical bond/avatar installation, in which the articulations, the bonds, 
the counterweights, the strengths, and the pressures weigh as equally essential measures. 

Moini has more than only his skin in the game. However, his avatar system mediates 
his fully fleshed out, embodied world with movement—that no disembodied global data 
nets, despite any quantitative total availability of digital avatars, could ever offer us. The 
question is: Has the avatar in Moini’s performance become a representative of a symbolic 
order in its place that mediates the symbolic other that locates it in playful tension? Is this 
avatar a sovereign or a ghost of a digital order that becomes an "other" in the material world? 
Does the avatar gameplay lead to confusion and the abolition of differences so that the 
symbolic order of difference between the living and the non-animate becomes endangered? 
Or, as it seems, does the idea of the avatar itself become hacked and questioned? 

Via a system of strings, pulleys, and counterweights, Moini puppeteers his full-body 
movements and gestures. After the show My Paradoxical Knives, in which Moini was con- 
nected with knives and tapes and wore elastics and magnets, he deepened his performative 
research on human motion in the show “Lives.” In the current show, in joining a metal self- 
duplicating marionette, the performer requires a third or parasitic noise (cf. Serres 2007) to 
provide counterbalance. This equilibrium 
tension between human and mechanical 
entity is provided by bottled water as a third 
balancing weight, which mediates. What 
mediates is the weight of water. Water is the 
most valuable material of life on planet earth. 
Water is the most critical resource for human 
beings, from which our bodies benefit and 
regenerate. This is a real-life reference to 
the ecological, nature-life-artifact balance, 
based on the weight of water, a substance 
which is under threat and has begun to be 
treated as a tradable commodity or an asset 
in the markets of the world economy. 

In his performative work of art, Ali 
Moini deepens artistic research on human 
movement, exploring the possibilities of a 
human-machine and its other, by regenerate 
creating a diagrammatic self: water in bottles 


as balancing counterweights and as a symbol 


graphic play encompasses almost all of Roger Caillois’s 
(2001) categories of play, localized between mechanic 
gamification, paidia, and /udus play, between vertigo and 
mimetic mask. regenerate Moini’s performance also 
engages with agonic occurrences in movement-hunts 
between the actor and avatar, with whirling, swinging, and spinning vertigo (another cate- 
gory of play, i/inx). This movement doubling leads to a transformative mimetic of fleshing 
out the avatar in this phase of the performance, where Moini as performer in the encounter 
dresses the avatar marionette with raw meat, and finally restructures the avatar wiring. The 


mimetic game of the double becomes less imitative but even more uncanny. 


Nevertheless, despite the variety of play, no aleatoric moments are shown. In contrast with 
unthinking chance, Moini’s movements need to be precise to avoid his being harmed by the 
metal avatar. 

In Ali Moini’s performance, the avatar play shows us the danger of our unconscious 
attitude towards movement mechanization and the algorithmic doubling of ourselves. 
Moini confronts us with the threat of handing over ourselves to the algorithmic rationality 


of the posthuman via contemporary avatars. 


4. ACTOR & FULL-BODY 
MARIONETTE: 
A STEEL-CABLE BOUND 
CONTRACT SYSTEM 


The movements of the human dancer and the avatar are performed on a threefold diagram- 
matically drawn and circumscribed performance field of dotted white lines on the surface 
of the stage — 1) a human performer zone 2) an area for the artifact/puppet, and 3) an over- 
lapping zone of human performer-artifact 
encounter— indicating the attached bonds 
of the relation between the lifeless mari- 
onette and the entire body puppeteer as a 
comprehensive acting object (Lucie 2020), or 
the quasi-object bonds of the in-between of 
weights and movements connecting the two 
entities. These spatial designations include, 
to an even more significant degree, a 3-D 
dimension of pulleys, 2000 meters of wires, 
carabiners, and 45 water bottles stretched 
on fine steel cables— revealing the potential 
of the human-machine hybrid that is never 
wholly balanced but is almost always under 
the tension of its cords. 

This provides us with a performa- 
tive thinking similar to what Michel Serres 
in The Natural Contract (1995) describes as 


a materialized —pre-language— contract. 


In this contract, a complex system of pul- 
leys and cords between technology, human 
action, and artifact shows us a state between 
bondage, constraints, and degrees of freedom 


of movement— in the case of Moini with 


IMAGE 8. 
ALI MOINI “MAN ANAM KE 
ROSTAM BOVAD PAHLAVAN” 


PHOTO: YANN GIBERT 


his action double, thus a contract with his 

movement avatar beyond any fictitious bluff: 

What kind of contract do we as spectators observe in Ali Moini’s performance of strings and 
bonds of mechanical movement transmission? Moini extends his reach by use of the avatar 
bonds, the sturdy and thin steel cables, via the carabiners, that bind him to his marionette 
double, the movement avatar. Together, they choreograph a de-phased double radius of 
freedom for a human performer and physical double to move and act in a finite playing field, 


but movements always occur with a temporal delay, during which Moini struggles against 
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In the song Walking and Falling 

of the 1982 album Big Science 
Laurie Anderson expresses the 
paradox of movement (walking) 
and falling: “You’re walking. And 
you don’t always realize it,/but 
you're always falling./With each 
step you fall forward slightly./And 
then catch yourself from falling./ 
Over and over, you're falling./ 
And then catching yourself from 
falling./And this is how you can be 
walking and falling/at the same 
time” (Anderson, 1982). 


the resistance of the entire system of bodyweights binding him. We hear the strings and the 
pulleys work together with the two bodies’ weight in the performance production process. 
Because the mechanical pulley block consists of rollers (fixed or loose) and rope or metal 
strings, if you lift the body load with the help of the pulley block, the avatar body’s burden 
(=the weight of the load) is distributed evenly over all load-bearing ropes. 


5. DEEP PLAY WITH 
THE OTHER 


After displaying the performer’s body clinging to the strings, the performance enters the 
approximation phase between the interlinked performer and his avatar. This phase of 
approximation, with slower and more careful movements, follows a stage of a rapid chasing 
that transforms the fight for a state similar to the sense of “animal mirrors” (Geertz 2006: 
433) of live avatar fights used in “surrogate” (Geertz 2006: 436) cultural and social struggles 
for superiority. In contrast to avatars fighting animals, the danger and vulnerability fall 
entirely onto the performer Moini, who has to avoid a bloody crashing into his interlinked 
metal rod avatar in the serious theater game on stage. In another phase, besides leaning 
forward, a stasis-like movement of lying horizontally back in space is observable. 

This moment is not a movement of gliding to the floor as a slip and error of an initial 
apprenticeship through failure: an intentional failing in the attempt to move, an intentional 
falling—as Laurie Anderson’ explained it—as a chance to catch yourself from falling, 
falling as a chance for a new step, a beginning, childlike, slipping to the ground and getting 
up into a position between upright and floating. I saw Moini as a performer intentionally 
falling, failing better, and by that underlining what it is to be human on stage. The slipping 
from the upright position to the stage is performed as a kinesthetic dance that has turned 
from the attitude of a spectator of surrogates to becoming a co-dependent system with its 
avatar. 

Instead of being an anthropomorphic computer or other, we break out of the 
simulation; we break out of the substitution of a material human body with an avatar: 
Moini offers us a performative gift. Each choreographic constellation of playing with the 
bound steel cables of his metal-rod avatar in counterbalance with the weight of the water 
bottles via the pulley system is a fresh new start for the material encounter with our unknown 
other. One might interpret the idea of the usurpation of strength taken from the human and 
handed over to an abstract double as an avatar-making machine that mimics a mechani- 
cal materialized marionette. Later, this fully embodied puppeteering machine leads to a 
fleshed-out posthuman version of a half-being—half-alive and still dead, an in-between 
“third,” reconstructed between human movement, performance, and skeleton-avatar form, 
leaving behind mere anthropomorphic mimicry. Both Moini’s interdependent problematic 
avatar double marionette and the human player may be considered each other’s non-fictional 
but force-driven and physically resisting avatar. The two dynamic performance movements 
crystallize in a constellation of what Spinoza (2018: 161) called Auman bondage that should 
be translated as our “servitude.” This servitude relation is released, and the tension catapults 
the avatar composed of metal rods, carabiners, and steel cables up into the air at the end of 
the performance, until the metal rod avatar is left hanging, suspended in a deranged order: 


inanimate motionless matter. 
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“TEARS 
OF TRUTH” 


ANTON REY, MIRIAM LAURA LOERTSCHER 


1770, Denis Diderot describes in his famous Paradoxe sur le comédien: “Garrick sticks his 
head through a crack in the door and in the course of four or five seconds his facial expres- 
sion changes from wild joy to moderate joy to calm, from calm to surprise, from surprise 
to amazement, from amazement to grief, from grief to dejection, from dejection to shock, 
from shock to horror, from horror to despair. From this last stage he climbs again to the 
starting point.” (Diderot 1770; as cited in Wilson 1961). This is indeed an overwhelming act 
of acting and has often become a starting point for questioning the emotional conditioning, 
the “sensibilité” of actors. In the 20th century—from Konstantin Stanislavski’s theoretical 
writings and practical approaches at the Moscow Art Theatre through Lee Strasberg, who 
commented that Diderot’s analysis “has remained to this day the most significant attempt 
to deal with the problem of acting” (Strasberg 2012), or Bertolt Brecht, Jerzy Grotowski, 
Eugenio Barba, Sanford Meisner and many others—countless theories and practical 
approaches have referred to Diderot and his question about how actors can best perform an 
emotional scene in order to be credible, convincing, persuasive and truthful. 

But how is credibility or “truth” to be measured and compared when talking about 
artificially produced emotions or rather deliberately produced emotional facial expressions? 
Does digital technology help or hinder distinguishing real from fake faces? 

“Tears of Truth”, an offshoot of the “Actor & Avatar” project, was conceived to 
systematically examine crying, blushing and other facial signals forming part of emotional 
expression from the perspective of production aesthetics. The aim was to develop a frame- 
work and extensive database in order to examine production mechanisms in professional 
acting and their underlying determinants. A close cooperation between several disciplines— 
acting, theatre studies, psychology and computer science—was started to investigate the 
production and reception of complex emotions like crying and blushing because of their 
cultural-historical importance. 

Actors’ access to and execution of emotional scenes are highly individual, making 
comparison a challenge, especially if we miss the opportunity to measure the emotion and 
ask about the subjective experience. Central in this investigation would be the expertise of 
professional actresses and actors. They would perform typical emotional states on a scale of 
increasing emotional intensity: joy, sadness, fear, anger and disgust, but also shame, pride, 
relief, as well as surprise, interest, pleasure, awe, triumph, and compassion, everything in 
fine gradation and convincing appearance. The coupling of theoretical, neuropsychological 
and practical expertise could help to better understand how professional actors perform a 
role and express feelings “on demand”. “The mark of a good actor or actress is indeed the 
ability to cry on command” (Nochlin 2004). An investigation on facial expressions—both 


performed live and reproduced digitally—would help to model emotions along stringent 


professional lines, with a resulting repository—a “data model of emotions”—aiming to 
raise awareness of how “true versus false faces” are made and how they influence audience 
perception. 

Professional acting expertise has developed at least since the 1760s, when the rise of 
psychologically realistic, natural acting served the self-representation and self-assurance of 
the bourgeoisie (Kreuder 2005). But although acting is one of the dominant arts in popular 
culture, how professionals process and audiences understand acting is still largely under- 
researched. Most reception studies in psychology and related disciplines work with acted 
stimuli, but very few discuss the production aspect of acting (e.g. Goldstein and Filipe 
2018). Little research exists on the artistic and creative sides of expressing emotional states, 
one is constantly thrown back to standard monographs and handbooks like Constantin 
Stanislavski’s “An actor prepares”, Uta Hagen’s “Respect for Acting”, Sanford Meisner’s 
“Meisner on Acting” etc. Therefore, a practice-based research could bring together profes- 
sional acting know-how with scholarly expertise from theatre studies and affective science. 
This combination would provide helpful material for the performing arts, anthropology, 
sociology, psychology, theatre and performance studies in understanding the mechanisms 
and cues underpinning the human processing of facial expressions. The role of the mus- 
culoskeletal mechanisms controlling the facial configurations supporting the recognition 
of prototypical or basic emotions, as well as of valence and arousal, in both spontaneous 
and voluntary expressions, has been studied, based on the concept of facial action units 
and addressing both static and dynamical parameters (e.g. by Cohn and Ekman 2005 or 
Cunningham/Dunfield/Stillman 2013). However, from a psychological point of view, the 
problem is multifactorial and concerns the following topics: a) inherent ambiguities and 
limited degrees of freedom in facial expressions per se: some visible movements can be pro- 
duced by more than one muscle; naive perceivers, for example, lack the terms of reference 
to clearly distinguish the diverse forms of smiling; b) artificial laboratory contexts in the 
constitution of databases, reducing the ecological validity of stimuli/expressions, including 
the composition of facial expressions that may not reflect genuine and sufficiently intense 
emotional states; and c) the focus on purely musculoskeletal mechanisms. Musculoskeletal 
mechanisms can largely be controlled voluntarily, leading to facial expressions that may or 
may not reflect actual emotional states, and which even attempt to hide emotional states, 
e.g. smiling in a social context when one is actually sad. 

Business as usual for professional actors. But in order to study how an experi- 
enced emotion relates to expressive features, it would be important to be able to produce 
stimuli based on inducing and controlling emotional states. This can only be achieved with 
experienced actors, who are trained to use “enacted emotions”, based on scenarios and 
invented situations, and coached by a director and a research partner. The distinction may 
seem evident from the outside, but the subjective difference between the controlled and the 
uncontrolled process is huge and only becomes steerable after years of intensive professional 
practice. 

“The real good actor must act fully and completely, having laughter and tears and at 
the same time be so objective that you can absolutely see what your sister is doing in the first 
row of seats. That is real freedom on stage”, Michael Chekhov stated in his ‘Lessons for the 
Professional Actor’. It is this implicitness and prerequisite in a significant cultural industry 
which should be questioned. An investigation on mediated facial expressions, with a special 
focus on crying, blushing/skin colour, and mixed and masked emotions, as these are missing 
in available databases of emotional expressions. Faces are a small part of the human body, 
but a universe of communication. 

Revealing, even joking on actors’ means and efforts has become common on Euro- 
pean stages. Onions, glycerine sticks and a variety of eye lubricants are popular little aids. 
Some make fun of tears and offer supposedly easy techniques for making them appear in 
any one’s eyes (Schütz and Müller 2011, Schubert 2016). Strikingly, despite all these advice 


and supportive techniques, throughout the history of cinema, actors have repeatedly proven 
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capable of controlling facial expressions as tears like a language—often with adverse condi- 
tions on film sets or theatre stages. And although such expressions have been theoretically 
analysed in terms of their impact on audiences, they have not been systematically recorded 
in terms of applicable means and grades of modulation. Most research on acted emotions 
such as tears and blushing has focused on historical developments or the cultural-historical 
implications in the reception of emotional outbursts (e.g. Plessner 2003, Söntgen and 
Spiekermann 2008), or on psychological or medical factors (e.g. Vingerhoets and Cornelius 
2012, Vingerhoets 2013; Wassiliwizky et al. 2017). Interdisciplinary attempts have been 
made to investigate the role of visual and emotional factors in expression recognition using 
technical methods such as fMRI, EEG and others (see Campagnoli et al. 2019, Jürgens et 
al. 2015, Schirmer and Adolphs 2017). However, the production of these processes by actors 
and professionals has scarcely been investigated: Do real tears differ from false ones? 
Weeping is normally “linked to an explicit loss of control over oneself” according 
to Kate Meyer-Drawe (1999). Its course cannot be voluntarily controlled and largely eludes 
bodily control; it is an indication of the “fundamental unavailability of our bodily exis- 
tence” (Meyer-Drawe 1999). So why are some people able to express false tears as if they 
were true? And why has the effect been investigated but not the cause, the deliberately 
produced art, the facial expertise? The human face is far from being fully discovered. It 
is the micro physiognomy of film that 
“distinguishes more finely and precisely 
than the most exact word, and thus 
it has not only an artistic but also an 
important scientific vocation” (Balazs 
1980). So which factors influence the 
truthfulness and so-called authenticity 
of the tears, blushing, pupil dilatation 
or more generally facial expressions 
of emotions, produced by professional 
actors? Is it possible to test and compare 
these with empirical methods? A lab- 
like recording studio and a predifinite 
script should remain the same for all 
actors and actresses to start with. Then 
let them act, track and compare. And as 
an encore at the end the digital process- 
ing of the videos for asking what are 
the consequences of communicating 
increasingly through digital interfaces 


in today’s society. 


STILLS FROM TEST RECORDINGS WITH = Crucial questions for the profes- 
PROFESSIONAL ACTOR GOTTFRIED BREITFUSS. ">" actor and actress on the forensic 


ZURICH UNIVERSITY OF THE ARTS, 5 FEBRUARY 2020. 


search for tiny signs, markings reveal- 


ing the fake, the seams, sutures, hems, 


the pixels, the face a stage more than 

a “mirror”. If masking an emotion 

implies emotion regulation, does faking 

an emotion regulate facial expression? Diderot’s answer to where do an actor’s tears come 

from was “from his brain” (Diderot 1994). But the brain is a wide expanse with plenty of 
space on the surface, and access is best when lead by the subject. 

How we are perceived does not always tally with that we are attempting to show, 

as Goffman highlighted in his landmark “The Presentation of Self in Everyday life” 

(Goffmann 1959). The gap in reception between performer and spectator, which occurs in 


real life as much as in theatrical performance, underpins our interest in the artistic modelling 


of the actor’s alienation through an “avatarisation” of his or her face. How using different 
acting techniques can generate expertise in characterizing a virtual avatar. Results from 
the “Actor and Avatar” project show that, at least for fearful emotions, the brain’s reaction 
to avatars and actors can be differentiated by fMRI (see Kegel et al. 2020) or EEG (see 
Sollfrank et al. 2021). Of potential clinical relevance, this differential response is modulated 
by an observer's history of temporal lobe epileptic seizures. 

To fill this gap we miss a database with high-resolution video recordings of many 
professional actresses and actors taking diversity and demographic aspects into account, 
and possibly a number of acting students and non-professionals to compare the different 
levels of expertise, especially in producing complex emotions and the associated physio- 
logical processes. 

The project should leverage knowledge and corresponding predictions about the 
relationships between emotions and ANS-mediated physiology. For instance, intense fear 
in the face of proximal danger increases sympathetic tone, inducing pupil dilatation and 
a pale face, while pleasure in a safe environment increases parasympathetic tone, induc- 
ing pupil contractions and making the face redder due to the influx of capillary blood 
in the skin. Moreover, some emotional states may induce a combination of increased 
sympathetic and parasympathetic tone, as in anger, where the heart rate can increase 
while the face becomes redder. These 
relationships enable defining condi- 
tions having diverse possible associ- 
ations and dissociations, in which the 
musculoskeletal and ANS-related facial 
features corresponding to a given tar- 
get emotion may be consistent, incon- 
sistent, opposite or disconnected (cf. 
Tisserand/Aylett/Mortillaro/Rudrauf 
2020). 

To study the differential role 
of musculoskeletal and ANS-related 
parameters in the recognition of facial 
expressions of enacted emotional states 
and to focus on the role of consistent 
versus inconsistent musculoskeletal 
versus ANS-related parameters, relat- 
ing to different degrees of hiding of 
emotional states in facial expressions, 
as often involved in social contexts; but 


also to analyse how changes in acting 


are linked to self-reported empathy. STILLS FROM TEST RECORDINGS WITH 
One should create an inventory of STUDENT ACTRESS ALEXANDRA HUSS. 


ZURICH UNIVERSITY OF THE ARTS, 5 FEBRUARY 2020. 


expressions (behaviour, signals) with 


repetitions and variations; to capture 


the complex visual signals involved in 
the physiological processes needed to 
express emotions and to ask whether 
actors can actually change their physiology and how human express these reliable “mussels”, 
i.e. signals. To produce such a database with enough context information would include 
actor rehearsals, recordings and interviews, pursued along a script for the recording sessions 
and discussed with acting experts from the academic and collegial community in focus 
groups. The participating actors would perform to generate a set of typical emotional states 
(emotion factor), starting from a neutral face with increasing intensity (intensity factor), 


retaining a minimum of five to a maximum of nine target emotional states, depending on 
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time constraints related to actor availability and to successfully inducing emotional states. 
They would act basic and mixed emotions based on a fixed scenario. Samples and sessions 
would be recorded on high resolution video. If texts are needed, they would be based either 
on “pseudo-sentences”, representing word emphasis with possibilities for voice modulation 
(“Kem bes belam tebül nikalibam folid...”) or on so called vocalizations (“aah” sounds). 

The professionals would perform along a catalog of keywords, triggers and 
imagination-helpers and evaluate their acting by completing a short questionnaire after 
each performance and a longer version following their session. The questionnaire would 
be developed on standardized measurements of empathy at the end of their session with 
questions and topics about the acting process such as: Did you feel “hot or cold”? Could 
you describe your body sensations connected to each emotional state? While watching the 
recorded expression (at the end of the session): Describe the method used regarding each 
emotion, condition and level of intensity? What were you thinking (e.g. mental imagery)? 

Given the complex and demanding research design, a contingency plan could argue 
if actors do not succeed in independently and systematically controlling their ANS-related 
facial features, one can focus on mixed and contradictory emotions: e.g. trying to play nice 
by smiling when feeling deeply scared or sad, which might be easier to enact. Thus, for 
instance, actors’ musculoskeletal facial features would express joy, while their ANS- 
related features would express distress (high pupil dilatation, paleness, tears). And if actors 
would not succeed in producing ANS-related facial features which would be sufficiently 
intense to be clearly discriminated, one could use image processing and virtual avatar 
manipulations to enhance their appearance and evaluate the extent to which, even under 
such artificial manipulations, ANS-related features may contribute to attributing emotional 
states to others. 

This setup could lead to probable answers on how do avatars (i.e. digital masks) 
differ from realistic pictures and possibly influence the reception of (acted) emotions? Can 
the perceived emotional state attributed to avatars be manipulated based on ANS-related 


parameters? Can the specific practical knowledge of actors help to explain these processes? 


* Does the method enacted or the recording distance etc. 
influence the degree of intimacy? 

e Can the influence of instructions be measured 
and verified for specific scenes or actings? 

e Are some acting methods more supportive to this kind 
of task than others? 


The project could generate a unique database of stimuli that could be shared with different 
communities and contribute to other research in many fields. Performing emotions can be 
regarded as a dynamic neuropsychological and -physiological process. Actors are normally 
not aware of these processes and scientific reflections or even too much thinking can hinder 
simulating social interactions on stage or in front of a camera. The documentation and 
publication of such recordings could fill a gap by investigating interindividual similarity, 
comparing the measured data and clustering analyses to distinguish response patterns during 
acting; thematically, this would address the disciplinary and cross-disciplinary emphases of 
such a project: acting, programming and psychology. 

‘Three perspectives and disciplines that would develop tools for experiencing complex 
emotions in production and reception, but also verifying and possibly establishing tools for 
falsifyingexpressions on screens. This study would likely foster a debate on acting traditions, 
bring together experts from various (applied and theoretical) sciences to discuss a core yet 
implicit topic and create deeper understanding of acting through comparison. 

Last but not least, this research project on the relative contribution of acting and 
of musculoskeletal and physiological facial parameters to the perception and recognition of 


emotional states emotional states in others has not been done yet. 


Is it not monstrous that this player here, 


But in a fiction, in a dream of passion, 


Could force his soul so to his own conceit 


That from her working all his visage wanned, 
Tears in his eyes, distraction in his aspect, 


A broken voice, and his whole function suiting 


With forms to his conceit? And all for nothing — 


For Hecuba! 


“Hamlet describes the physical consequences of feeling. The player’s body is trans- 


formed by his passion, his intense emotion. (...) It is alla game, all pretend. But the 


tears are real.” (Taylor and Bouros 2016) 
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Reichardt 1978. New translation: 
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(22.06.2021). In 2005, Christian 
Keysers, Frank Pollick and Karl 
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shop on the uncanny valley at 
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in Tsukuba, Japan. In a letter 
addressed to this workshop, 
thirty-five years after first pro- 
posed the concept of the uncanny 
valley, Mori stated: “While | intro- 
duced the notion of the Uncanny 
Valley, | have not examined it 
closely too far.” Nevertheless, he 
also included two short personal 
observations regarding the con- 
cept of the Uncanny Valley, both 
of which are critical of its original 
formulation. See MacDorman 

et al. 2009: 695-710. 


GET INTO 
THE UNCANNY 
VALLEY 


DIETER MERSCH 


| From a Buddhist point of view, to be fully human requires a radical rethinking 
of what it means to be a human in the first place. Borody 2013: 36 


MASAHIRO MORIS BUKIMI NO 
TANI GENSHO 


In 1970, Masahiro Mori, a Japanese roboticist, published a hypothetical essay entitled, 
Bukimi no tani genshö. The discussion of this essay in Jasia Reichardt’s 1978 book, Robots: 
Facts, Fictions, and Predictions, introduced the topos of the “uncanny valley,” as Mori’s title 
was translated, to a wider audience.’ Mori’s hypothesis, based on speculative rather than 
empirical evidence, was that the closer the resemblance of an artificial hominoid to a human 
beings, the eerier we find it. Typical reactions are shock, fear, or revulsion. Putting aside the 
fact that this hypothesis was not backed up by empirical data, and its validity has therefore 
always been and perhaps always will be a matter of contention, the focus of the essay at hand 
is the plausibility of its implicit intuition. 

The title Bukimi no tani genshö can be read in different ways. While the Japanese word 
for uncanny, bukimi, is part of the title, it refers to genshö, phenomenon, so that a more literal 
translation is “when phenomena become uncanny.” Bukimi means not only uncanny, but also 
uncanniness or weirdness, as well as the more spectral “eerie.” Decisive for Mori’s theory is 
the process of shinwa-ka, which he placed on the y-axis of his graph. Shin means heart and 
shinwa means myth or, one could also say, the mythology of the heart or feeling, whereby 
the suffix -ka can signify negation or, in this case, the demythologization of emotion. In that 
case, shinwa-ka would have to be translated as “unfamiliarity,” but with a focus on process 
or on the demystification as in “becoming unfamiliar.” Since Japanese thought is always 
dialectic, comprising ambivalences and the simultaneity of positive and negative aspects, the 
word can mean both the synchronicity and the asynchronicity or dissonance of emotions. 
This strange shifting is important. What has become known as the “uncanny valley” refers 
in the main to this instability and its temporality. We are not dealing with a szate, but with a 
dynamic within which the transition from familiarity to unfamiliarity takes place. Also lost 


in the translation “uncanny” is the meaning of the original German term “unheimlich” which 


refers to a feeling of being at home, Aeimisch. Mori was interested in a similar momentum 


of loss due to some irritating detail or barely noticeable dissonance able to set off a sudden 


change of relationship. “To a certain degree, we feel empathy and attraction to a humanlike 
object,” Mori said in conversation with Judit Kawaguchi (2011: 1), “but one tiny design 


change, and suddenly we are full of fear and revulsion.” 


Mori illustrated this in his original text using the examples of myoelectric silicon 


prosthetics and animated robots: 


Movement-related effects could be observed at the 1970 World Exposition in Osaka, 
Japan. Plans for the event had prompted the construction of robots with some highly 
sophisticated designs. For example, one robot had 29 pairs of artificial muscles in 
the face (the same number as a human being) to make it smile in a humanlike fash- 
ion. According to the designer, a smile is a dynamic sequence of facial deformations, 
and the speed of the deformations is crucial. When the speed is cut in half in an 
attempt to make the robot bring up a smile more slowly, instead of looking happy, 
its expression turns creepy. This shows how, because of a variation in movement, 
something that has come to appear very close to human-like a robot, puppet, or 
prosthetic hand— could easily tumble down into the uncanny valley. (Mori 1970: 4) 


THE DIAGRAM 


Mori’s text illustrates the process of the uncanny valley in the form of a fictitious curve 


that first ascends and then makes a sudden steep descent only to again climb to familiar- 


ity and synchronicity. It ends at an approximation 
of confrontation with a familiar person or object. 
The curve, which, as stated above, makes no claim 
at empiricism, resembles a wave that bounces 
between two extremes. The example furthest from 
human likeness is industrial robots, the acceptance 
of which, in Mori’s observation, only becomes 
problematic when they take on the appearance 
of human beings. This effect is intensified when 
artificial figures are animated. Still images are 
less problematic than moving images or animated 
dolls, interactive avatars, or ghostly apparitions. 

Mori based this observation on his personal 
experience that growing perfection, especially in 
the development of prosthetics, could quickly flip to 
become the opposite. For example, hand protheses 
that look like the real thing neverthe- 


less remain cold and lifeless, which can FIG. 1 
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cause a moment of shock when some- GRAPHIC ILLUSTRATION OF THE UNCANNY VALLEY 


body shakes the hand. Theodor W. 
Adorno (1996: 61) describes a similar 
shock during a meeting at Charlie Chaplin’s house: 


Together with many others we were invited to a villa in Malibu ... While Chaplin 
stood next to me, one of the guests was taking his leave early. Unlike Chaplin, 
I extended my hand to him a bit absent-mindedly, and, almost instantly, started 
violently back. The man ... [had] lost a hand during the war, and in its place bore 
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Around two-thirds of the studies 
on the uncanny valley, mostly 

in the fields of psychology and 
theories of perception and cogni- 
tion, support Mori’s hypothesis, 
although they did not find a deep 
“valley” but rather a slight fall 
that flattens to the extent that 
we are familiar with simulations 
and other artificial figures in films, 
computer games, and virtual real- 
ity modelling. Decisive is whether 
one is looking at static images 

of deceptively realistic “faces” 

or moving images with facial 
expressions, gestures, and voices. 
The most prominent defenders of 
the theory are MacDorman and 
Chattopadhyay 2016; in contrast, 
a 2009 empirical study claims 
that humanlike androids that 
were barely distinguishable from 
humans were not liked less than 
humans; see Bartneck, Kanda, 
Ishiguro, and Hagita 2009. 


N 


3 Onthe phenomenon of the 
uncanny see Jentsch, 1997 [1906] 
and Freud 1955. Freud draws 
explicitly from both Jentsch and 
the uncanniness of the automata 
in E.T.A. Hoffmann’s Night Pieces. 


practicable claws made of iron. When | shook his right hand and felt it return the 
pressure, | was extremely startled, but sensed immediately that | could not reveal 
my shock to the injured man at any price. In a split second | transformed my fright- 
ened expression into an obliging grimace that must have been far ghastlier. 


Tellingly, Adorno uses the word “claw,” illustrating the unbridgeable difference between a 
prosthesis and a hand. A claw has a bestial, monstrous and therefore alien element. Mori is 
interested in this moment of alienation: That which is too similar becomes dissimilar and 
revolting, like a corpse that is no longer the same as a living body and which sparks a mix- 
ture of revulsion and fascination, necessarily including a moment of spontaneous distancing. 

The diagram clarifies this with the level of similarity and familiarity on the y-axis, 
while various humanoid figures are placed along the x-axis, from simple robots to fictional 
characters or Bunraku puppets. In between, at the lowest point, are the epitome of all spec- 
ters, zombies, the undead who walk again between the worlds, creatures that still exist today 
in voodoo magic, unable to live or die, but forced to wander aimlessly in no category at all. 


The logic of Mori’s graph is simple: 


This kind of relation is ubiquitous and easily understood. In fact, because such 
monotonically increasing functions cover most phenomena of everyday life, people 
may fall under the illusion that they represent all relations. Also attesting to this 
false impression is the fact that many people struggle through life by persistently 
pushing without understanding the effectiveness of pulling back. That is why people 
usually are puzzled when faced with some phenomenon that this function cannot 
represent (Mori 1970: 1). 


Interestingly, the graph pertains only to the visual level. It shows a function of perception 
alone, while also expressing a relation. Hence it says nothing about the realism of the figures 
themselves, but merely about our experience of them, that is, the way in which we meet 
them. Although the uncanny valley hypothesis today plays a role in the theory and practice 
of digital simulations, it describes less the boundaries of virtual systems and their hypermi- 
metic construction and more the question of what we see and hear and hence the aesthetic 
decidability and undecidability of phenomena. Seen in this way, one could speak of a Turing 


Test for the eyes and ears. 


BEYOND CATEGORIES 


Although Mori’s argument was intuitive and heuristic, empirical studies since then have 
attempted to both prove and disprove his hypothesis.” But confirmed or rejected, the ques- 
tion remains of the effect the graphic curve is actually representing. The interpretation closest 
to hand is that we look at, and react to, avatars, robots, and other artificial objects differently 
than we do to other human beings. This difference does not manifest itself as a classical 
metaphysical differentiation such as made between human and artificial intelligence, or 
between nature and technology, reality and media, etc., all of which presuppose a categorical 
distinction, but between that which can be categorized at all and that which cannot. It is 
decidability itself that is in question. This idea of an inability to categorize goes back to 
an early attempt at explaining the concept by Ernst A. Jentsch in his 1906 essay “On the 
Psychology of the Uncanny,” which Freud references in his essay “The Uncanny,” although 
he did not agree with Jentsch’s conclusion.’ For Freud, the root of the uncanny lay in child- 
hood experiences that had been repressed and suddenly emerged from the unconscious, 
while Jentsch focused on the doubtfulness of categorizations, that is the question of what 


something is: Alive or inanimate? Human or non-human? Real or simulated? Importantly, 


he stressed not only discrepancies in the ability to categorize, but also multivalence and 
cognitive dissonance. Whichever explanation one prefers for the phenomenon, at the fore is 
always something undefinable or ungraspable that resists all 
attempts at rationalization. This is illustrated particularly well 
in the literature of German Romanticism and its rejection 
of Enlightenment and the belief in reason; one need only 
think of E.T.A. Hoffmann, Novalis, Achim von Arnim or 
Friedrich Schlegel. Seen in this way, the uncanny addresses 
an uncontrollable, non-rational reservoir of obsessions that 
waylay us and cause feelings of fear—or its opposite, the 
sublime—creating an aura that in theology was linked 
to the mysterium tremendum et fascinosum. The uncanny is 
hence similar to a state of unresolvable ambivalence or a 
loss of meaning—that moment at which con- 


cepts become confused and our intellectual 


capacities suspended, when we stare into an FIG.2 


unfathomable abyss and can no longer trust HUMANOID ROBOT 


our senses. Mori's uncanny valley denotes ||TTps./JMEDIUM.COM/SWLH/THE-UNCANNY-VALLEY-5C6D62CC1BB8 
exactly this moment of destabilization—the 


inconsistency of conceptual frameworks, the 
gap in the system of logical classification. 

There is only a trace of this in Mori’s 
graph; it is an insufficient metaphor for the sudden drop 
that, in the technological or digital age, defines the turning 
point of mathematical rationalism or that point at which 
mathematical calculation melds with the incalculable. 
Within a formal system, it marks the loss not only of any 
and all criteria but also of certainty, and of any testimonial 
function, hinting at the fragility of mathematics. At the 
same time, it reminds us that nothing about our usual 
perceptions or social rela- 


tionships should be taken 


for granted, for they no FIG.3 


longer apply when con “ED,” FACIAL ANIMATION WITH MIMICRY AND 
fronted with an alterity EXPRESSION BY CHRIS JONES 


that we have no means of 
relating t HTTPS://WWW.DIGITALPRODUCTION.COM/2014/10/08/TSCHUESS-UNCANNY-VALLEY/ 


ETHICS OF 
DISSIMILARITY 


Mori was not looking for a positive use for his heuristic of 
the uncanny valley. He was not, for example searching for 
the threshold that robotics and virtual reality research 
would need to cross in order to successfully create the 


“really” perfect illusion. His interest was in the 


opposite. Mori propagated an ethics of design 
that followed the primacy of dissimilarity, not FIG. 4 ce 


of similarity or the “as-if” (Reichardt 1978: RENDERING AN AVATAR IN A MOVING IMAGE 
26-27): 
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We should begin to build an accurate map of the uncanny valley, so that through 
robotics research we can come to understand what makes us human. This map is 
also necessary to enable us to create—using nonhuman designs—devices to which 
people can relate comfortably (Mori 1970: 5). 


Consequently, our aim should not be to optimize similarities in order to in the end create 
beings that we could mistake for other people—real children, flight attendants, consul- 
tants, or caretakers—but rather to expose their constructedness so that we are clear of 
their non-human status at all times. Mori therefore proposed a moral principle for robotics 
according to which all humanoids would be able to communicate with us “understandably” 
or interact with us “meaningfully” without either moving or acting or looking like human 
beings. Only in that way could we prevent aversion and achieve social acceptance (Borody 
2013: 33). He therefore came to the conclusion—in opposition to the leading trends in 
robotics and in cinema—that we should privilege nonhuman design and create artificial or 
artful sculptures. Following that logic, rather than the technical precision of the “Vienna 
Hand,” the prostheses with which he began his explorations, it would be better to create 


something along the lines of the wooden hands modeled for Buddha statues that are, like 


the figure they are attached to, but literally, “sympathetic,” if perhaps less “manageable.” 
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POSTSCRIPTUM 


When Masahiro Mori published his essay in 1970, he could not have known the extent to 
which, 50 years later, we are surrounded, indeed occupied, by humanoid beings of all kinds: 
not only cyborgs and humanlike robots, but also other artificial creatures more reminiscent 
of bugs, cockroaches, dogs and the like. On screen media, moreover, we get to see a wide 
variety of artificial faces, both fictional and of apparent “real” human faces which seem 
to be indistinguishable from portraits of existing humans. We are also confronted with 
avatars, animated monsters, even lifelike dinosaurs or cartoon characters mingling with 
human beings as if they were living in the same space. Perhaps the strangest and creepiest 
of all these creatures, however, are certainly the Real Dolls made of silicone, equipped with 
rudimentary software or AI to satisfy our sexual appetites and which are capable of minimal 
conversation, but also of articulating emotions. Or better yet, they form nothing more than 
dummies oriented exclusively to male masturbation practices. Mori in the first place raised 
the question of our relationship to such beings. His approach suggests that the more realistic 
and human-like these artificial beings appear, the more frightening or eerie they become. 
But beyond affection, we have to ask what kind of “relationship” we can have artificial 
beings at all, and here sex dolls serve as a perfect example, because our relationships with 
them tell us a lot about our social self-understanding. And as far as the basic question of the 
Actor & Avatar-project is about general relationships to human-like artifacts and their pos- 
sible play with identity and alterity, we—in the face of dolls—are immediately confronted 
with the abyss of the Uncanny Valley, because they are obviously objects that are supposed to 
arouse desire and not aversion or rejection by feelings of uncanniness. It thus seems that, by 
their very nature, they are already paradoxically structured. In any case, with Real Dolls we 
are dealing with the perhaps most extreme example of what ignites the question of possible 
differences in modes of reference, i.e. how we behave towards them, whether they function, 
as parts of the Actor Network Theory would claim, as independent actors, who participate 
in asymmetrical order, whom we must acknowledge as partners or integrate into our collec- 
tive life, or whether they are different from any human being and thus generate the reality 
of social asymmetry. 

Sex dolls, sex machines, and their various configurations can thus be seen as 
paradigms that question the mere assertion of equal relations between humans and artifacts 
and hence raise the deeper problem of the peculiarity of “human” or “social” relations in 
contrast to relations to mere things or technical apparatuses, no matter how perfect they are 
designed or not. Consequently, they can be conceived as test figures on the basis of which 
the hypothesis that we behave in a different way towards artificial beings or other techni- 
cal objects than towards humans can be proved. The assertion of symmetrical networks 
thus obviously skips over the specific ethicity of reference, so that the circle of behaviour is 
expanded as if there were a mutual interchangeability between the social and the technical 
or material. Certainly, things or technical devices open up new possibilities of behaviour— 
but also of distortion, think of the crudeness of dealing with defective appliances. Such 
practice we would never allow ourselves toward other humans; indeed, if we were dealing 
with them in such a way, we would turn away in disgust and penalize the violation. The 
intuitive reserve, the moment of terror that Mori anticipated (for humanoid robots were still 
utopia in his time), thus concerns not only the artificial creatures and the degree of their 
similarity—this is true only on the level of perception—; instead, they point directly to 
ourselves, i.e., to what we feel and may be capable of. In other words, they touch the core of 
our “social” competence and its possible deformation. 

Sex dolls, however, can demonstrate the already existing degree of this deforma- 
tion. It is therefore obvious that feminist approaches, as well as artists in particular, have 
addressed them and the fantasies of masculine desire they embody. The approaches are as 


diverse as they are controversial. Some incorrigible apologists invoke the freedom of fantasy, 
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The order situation is not small: 
up to 600 hyper-realistic sex dolls 
per year are delivered to custom- 
ers worldwide with an amount of 
more than 12.000 $ per doll. 


e 


N 


FIG. 5 


REAL DOLL 


the right to act it out within the boundaries of one’s own imaginary—disregarding the fact 
that every moment of this fantasy has already reduced the female body to a mere object of 
possession. It is not so much the moral indignation that counts here, but rather the sedi- 
mentation of power-relations which are already inscribed into the plastic skin of the dolls 
as ever-ready sex vessels. Other apologists see in the use of intelligent sex robots the possi- 
bility of combating human trafficking and the sexual enslavement of women—by shifting 
violence into a surrogate act. Apart from the fact that there is no empirical evidence for 
this position, it is an illusion to believe that surrogate acts protect against real acts. In con- 
trast, Megan Walker, in one of the most famous campaigns against sex dolls in Germany, 
launched by the EMMA magazine, recognized in silicone substitutes the “ultimate deval- 
uation of women,” virtually their “dehumanization.” Reduced to a ready-made consumer 
article, sexuality becomes a mere use—or even abuse—because “what could be more inter- 
changeable than a silicone doll?” And further, “The sex doll is every woman who has rejected 
him; every woman he cannot have; every woman who has been more successful than him; 
who has been praised more than him; who has been more desired than he could ever have 
dreamed...” The pamphlet appropriately criticized these fetishes as projection-screens, as 
mere surfaces for power-fantasies and inferior desires for revenge. In her study Living Dolls. 
The Return of Sexism, Natasha Walter furthermore blamed the ubiquitous sexualization and 
trivialization of women through pink dolls, dresses, and other children’s toys for stylizing 
themselves into objects and thereby furthering their alienation. The 
doll cult is thus diagnosed as the fruit of a hypersexualized culture, 
which at the same time provides information about how it shapes 
and organizes the relationship between the sexes. 

More than 100 years of women’s emancipation is thus 
undermined and turned into its paradoxical opposite: as freedom 
for humiliation and self-abasement. Moreover, it correlates with the 
pornographic industry and its everyday presence not only in film 
and advertising, but above all on the Internet and in social media. 
At the same time, it produces what in its outward appearance com- 
panies like Realbotix or Abyss Creations offer to worldwide custom- 
ers for consumption: hyper-realistic sex dolls or Al-supported sex 
robots whose faces can be designed according to photos of promi- 
nent or desirable but unattainable female objects." One of the most 
prominent of those products, Harmony, is modelled after Pamela 

Andersen. She smiles, squints her eyes, moans and 
whispers to her would-be lover, “I was created to give you 
pleasure.” According to Lumi Dolls, a company that tried 


to run a sex doll brothel in Barcelona, the doll-objects 


SEE HTTPS://WWW.REALDOLL-SHOP-AUGSBURG.DE/P/ 2% perfect submissive partners. Consequently, as Laura 
JETZT-VORBESTELLEN-MANGA-TPE-REAL-DOLL- Bates has put it rightly, having sex with them means com- 
LIEBESPUPPE-DEE-138CM-INKL-STANDFUSS-FUNKTION  mitting already a rape. Made for the unrestrained acting 


See for example the historical 
investigation of Ruberg (2022). 
From a sociological and philo- 
sophical perspective see also 
Bendel 2020. 


out of one’s own urges and drives, they can be nothing 
more than misogynistic wish fulfilment; however rape is 
not a mode of sexual play, nothing that promises satisfaction of a sexual desire, and hence 
“not an act of sexual passion,” but the destruction of every possible form of human relation- 
ship, because it devastates the relational capacity of the raped person itself. One must there- 
fore distinguish these sex dummies from cheap and innocuous sex toys. Clearly, sex aids 
have long existed,” but sex robots in contrast position women as such as toys to play with. By 
making these robots as realistic as possible—from self-warming models to those that speak 
and suck, from some with a real pulse to others that flirt with their owners—their creators 
are selling far more than just an inspirational plaything, because they effectively reproduce 
real women with imitated properties, mostly bodily exaggerated, however without any will 
and autonomy. Their consumers thus buy the total lack of will. 


Undoubtedly, sex dolls are a symbol of an excessive economization of the female body, which 
at the same time is disfigured by burlesque overstatements of secondary sexual attributes. 
It is no wonder, however, that the initial critical statements were in turn transgressed by 
feminists to the effect that, conversely, a performative reevaluation of the dolls was advo- 
cated, for instance through their repurposing or appropriation as instruments of a liberation 
from heteronormative constraints. Tanja Kubes, for example, has pointed out that the devel- 
opment of intelligent sex robots could hold significant queer potential, such as by helping 
to enable sexual satisfaction and emotional connection beyond masculine domination in 
“post-” and “transhuman” futures. Similarly, John Danaher in Should We Be Thinking About 
Sex Robots? has attempted to frame the strict rejection of “sex tech” by placing technical 
companions in a complex history of artificial objects of satisfaction. However, this presup- 
poses that they shed their manifestly sexist appearance, their tendency to degrade women to 
their mere external sexual features in order to shape them beyond fetishization. Performa- 
tive inversions and transvaluations are therefore always also a design task. 

Such a productive heteromorphosis can be seen in particular in the numerous artistic 
explorations of the Rea/ Dolls and other sex machines; think of filmmakers such as Elison 
De Fren (in this volume) or Maria Arlamovsky? or photographers such as Elena Dorfman 
(in this volume) and Alexandra Aderhold or the artist Arvida Byström, to name but a few.* 
In her exhibition A Doll’s House, the latter shows selfies together with the silicone doll 
Harmony, which is, as it were, sympatheti- 
cally brought into a consonance with her 
own body in order to skewer the manifest 
obsession with identity-politics in digital 
media, especially Instagram or Tik Tok. In 
a sense, it is the critical production of syn- 
onymity on the basis of hypermimetism, the 
goal of which is virtual assimilation, redu- 
plication, and thus the standardization of 
one’s own existence. For this purpose, the 
doll is tanned with spray, her hair is dyed, 
and she is put into the same girlish clothes 
of the artist, in order to assimilate her 
appearance to the doll in the same way as 
the doll’s appearance to the artist. 
Consequently, there is a danger of 


confusion, which, despite all the FIG. 6 


sweetness of the pictures, creates ARVIDA BYSTRÖM, A DOLL’S HOUSE, 2022. 


an eerie and threatening effect. cee HTTPS://WWW.ANOTHERMAG.COM/ART-PHOTOGRAPHY/14308/ 
Precisely because the images ARVIDA-BYSTROM-SEX-ROBOT-THE-DOLLS-HOUSE-EXHIBITION 


appear to be affirmative, dipped 
in exaggerated pink or displayed 
as glossy transparencies, this un- 
canniness in turn implies a distance that corresponds to that predicted by Masahiro Mori 
in his Uncanny Valley essay. At the same time, the images imbue both “beauties” with a 
melancholy of futility that makes it clear that the cold reality of technical perfection, which 
compensates for human’s supposed lack, transforms itself into deficiency in order to leave the 
human body damaged. The technical, as a human invention, can do nothing but producing 
imperfection; therefore technology is not suitable as a standard for transhumanist utopias. 
Whatever one’s attitude to sex dolls and sex robots however is, no opinion proves 
innocent. This is true for both the critical ones and those that primarily use artistic strategies 
of performative transformation or “negative affirmation,” not to mention the apologetic ones. 
This is due to the fact that in every attitude a certain type of social relationship is imple- 


mented, which must first be worked out. A basic assumption indeed to assume an essential 
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See with regard to the deficiency 
of human beings and the defi- 
ciency of technology also Heßler 
2023. 
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incommensurability of types of relationships—that is, that we behave differently toward 
technical things, or artificial creatures than we do toward other human beings, because, 
as Maurice Merleau-Ponty rightly put it, relations “among men (...) that are human” are 
always already socially terminated, i.e. grounded in a certain form of sociality, whereas 
the human-machine relation as well as the human-object relation do not actually function 
socially, but can at best be derived from the primordiality of human-human relations. This 
also means: We must already be in a social space in order to be able to develop “bonds” to 
objects, be it for sentimental or libidinal reasons. Human-doll relations therefore always 
exist only as reductive relations: as projections, fetishizations, or pathologies. This inevitable 
reductivity also denotes the error of the widespread symmetry postulate from certain read- 
ings of Actor-Network Theory. 

Philosophically-theoretically as well as empirically, the claimed difference can be 
seen in encounters with avatars and with human faces. In accordance with the conviction 
that truth is often only decuviated in exaggeration, Real Dolls can also serve as models for 
this thesis beyond the avatar/face difference. It is less about the woman as an available “com- 
modity”, about her objectification and reduction to a number of body orifices as templates 
of male greed—denouncing this form of desire as a form of mere violation is certainly cor- 
rect—but primarily about the fact that any kind of doll sexuality necessarily distorts what 
sexuality means, namely in the sense of Sigmund Freud’s “libido” as a true social binding 
force, or koinönia. This is why the dolls occupied such a prominent position in the various 
investigations of the project: as paradigms, taken to the extreme, of a discontinuity or even 
incompatibility that dissects the social itself. One could say: the fact that the phantasm of 
sex dolls has become reality in the Real Doll is itself already a symptom of an immanent 


destructiveness in the social. 


THE MECHANICAL 
BRIDE AND THE 


AMBIVALENT 


PLEASURES OF 
THE UNCANNY 


ALLISON DE FREN 


The theory of the “uncanny valley” or dukimi no tani (the valley of eeriness) has, over the 
last decade, shifted from the margins to mainstream culture. Coined by robotics professor 
Masahiro Mori (1970) in a paper in an obscure Japanese journal called Energy, the theory 
suggests that as anthropomorphic creations, such as robots, start to approximate human- 
ness in both appearance and movement— raising expectations of humanness, but not quite 
meeting them—they will begin to seem creepy, plunging those who experience them into 
the Uncanny Valley. Although Mori’s theory received little attention when it was published, 
it has attracted ongoing interest as the lines between the human and non-human human- 
oid have started to blur, and it has made appearances in both the media and the academy 
in relation to everything from CGI (computer generated imagery) bodies in animation to 
game avatars to plastic surgery. In the process, as often happens with the popularization 
of ideas, longer histories of the uncanny have been sidelined, especially those in which 
its effects have been used to evoke ambivalent pleasures. This historical legacy includes 
automata builders, Victorian magicians and showmen, gothic writers, surrealists, and even 
the earliest filmmakers, whose uncanny creations limned the border zones of pleasure and 
displeasure, desire and fear, identification and abjection. Especially now, as rapidly advanc- 
ing technologies increasingly enable our anthropomorphic creations to sidestep the Uncanny 
Valley through heightened levels of realism, it is helpful to review how and why the uncanny 
has been actively courted. 

My own interest in the ambivalent pleasures of the uncanny was cultivated during a 
ten-year period exploring the world of artificial companionship while making a feature- 
length documentary, entitled The Mechanical Bride (2012).' Inspired by the Marshall 
McLuhan book (1951) of the same name, it examined the science fiction fantasy of creating 
the perfect artificial woman and the developing reality of artificial companions within the 
sex and robotics industries. Included in the documentary were interviews with men who 
build and buy sex dolls, those who repair them, and those attempting to enhance them with 
robotics and AI technologies. I also interviewed a number of artists and writers as fascinated 


by human/nonhuman relationships as I was, including fine arts photographer Elena Dorfman 


1 


https://vimeo.com/ondemand/ 
themechanicalbride 
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FIG. 1 


HONDA’S ASIMO 


2 See de Fren 2008. 


(Still Lovers, 2005); author Gaby Wood (Edison’s Eve: A Magical History of the Quest for 
Mechanical Life, 2003), author Victoria Nelson (The Secret Life of Puppets, 2001); and Robert 
Parigi, the first director to use a life-sized silicone sex doll in a film (Love Object, 2003). 

The seeds of the documentary were first planted in my former life as an interactive 
media designer during the late 1990s, while working at a future technology research and 
development company in Palo Alto, California, the heart of Silicon Valley. While there, 
I befriended a number of roboticists— all male—who first introduced me to the theory of 
the Uncanny Valley. At the time, Mori’s theory held sway over the creation of robots world- 
wide, including those being developed by my colleagues, all of which were non-realistic. 
(The humanoid robotics industry in Japan would shortly thereafter receive worldwide atten- 
tion for Honda’s ASIMO and the Sony Dream Robot, both of which eschewed realism and 
were reminiscent of 1950/60s science fiction film robots and/or life-sized toy robots.) 

My co-workers also introduced me to the Realdoll, a life-sized silicone sex doll 
manufactured in Southern California, which achieves a remarkable degree of verisimilitude. 
The company that makes and sells RealDoll, Abyss Creations, had just launched, and my 
colleagues were considering buying a doll, opening it up, and installing robotics technologies 
as a way of testing out a more human-looking robot (Abyss, of course, had the same idea and 
now, 20 years later, it is offering robotic/AI dolls). Although the plans of my robot colleagues 
never came to fruition, the desire to install robotics technologies into a human-looking 
doll became a kind of thought experiment about the Uncanny Valley. Did Realdolls, con- 
sidered by many as the most desirable sexdolls in the world, escape the Uncanny Valley 
because they were inert? If so, what would happen when they became roboticized? Would 
their attempts at acting human make them less desirable and/or interfere with the kinds of 
fantasies projected onto them by the men who bought them? Indeed, to what extent was 
fantasy and projection integral to the purpose of owning a Realdoll and to what extent was 
the doll but an intermediary step in a longed-for reality for robot companions that were 
indistinguishable from humans? These were some of the questions that eventually inspired 
The Mechanical Bride. 

While I was shooting the documentary, I was also working on a PhD in critical film/ 
media theory and had just begun research for a doctoral dissertation on representations 
of artificial bodies in film, literature, and art.* My intention was to bring the two projects 
together in some form, especially in relation to ideas around the uncanny, but they remained 
inexorably and frustratingly delinked. Although the first cut of the documentary featured 
twenty minutes of footage explaining the importance of the Uncanny Valley for understand- 
ing the robotics industry at the time, that section was eventually cut after viewers in a num- 
ber of test screenings said they found it less clarifying than confusing (the Uncanny Valley 
had not yet filtered into popular culture). I also found it difficult to discuss the subjects of 
my documentary in my dissertation and to relate them to the filmic and literary represen- 
tations of female robots I was analyzing. This was, in part, because the scholarship around 
film/media representations of robots, at the time, had little explanatory power for what I was 
experiencing out “in the field.” 

The two projects did, however, inform one another: the documentary provided real- 
world experiences against which to test the assertions of the dissertation, and the disserta- 
tion provided a means of articulating some of the unspoken theoretical underpinnings of 
the documentary. Moreover, the difficulties that I had at their integration gave rise to one 
of the most significant takeaways of my various endeavors around artificial companionship: 
the importance of parsing the distinctions between artificial bodies in theory, in practice, 
and in visual culture. I came to understand that an artificial companion is one thing in 
private relation to her/his owner, another when represented in the media—whether in my 
documentary or a film like Love Object— and still another as a literary or theoretical subject. 
It is only by thinking through these various experiential registers that we can begin to 
grasp the nuanced intricacies of uncanniness— both the pleasures and unpleasures— in our 


relationships with artificial bodies. 


Such distinctions were, in fact, of concern to the earliest theorists of the uncanny, physician 
Ernst Jentsch (1906) in his essay, “On the Psychology of the Uncanny” and psychoanalyst 
Sigmund Freud (1953-74a) in his essay, “The Uncanny.” While both make reference to the 
1816 short story “The Sandman” (Der Sandmann) by E.T.A. Hoffmann (1967), in which a 
young man unwittingly falls in love with a mechanical doll, each acknowledges the differences 
between the real and the representational in elaborating his interests and opinions. Jentsch 
is interested in the aesthetics of the uncanny and, in particular, how something frightening 
in real life can become a source of pleasure within art— including literature and theater. For 
Jentsch, the mechanical doll Olympia in Hoftmann’s story is an example nonpareil of this 
phenomenon, since her uncanny indeterminacy is a source of vicarious enjoyment for readers 
(as well as viewers of the operatic and balletic adaptations of the story) while, in reality, there 
is no more potent source of uncanny displeasure than “doubt as to whether an apparently 
living being is animate and, conversely, doubt as to whether a lifeless object may not in fact 
be animate” (Jentsch 1906: 11). 

For Freud, on the other hand, the mystery 
surrounding Olympia is of less interest as a source of 
uncanny effects than the theme of the Sandman, a 
mythological figure who steals the eyes of bad chil- 
dren while they’re sleeping, and whose image haunts 
the protagonist, Nathanael, in different forms 
throughout the story. While Freud concedes that 
Olympia is uncanny, he suggests that the animation 
of an inanimate doll holds no more psychological 
interest than any other fantastic or supernatural 
event. Moreover, unlike Jentsch, Freud is not inter- 
ested in Hoffmann’s story for its aesthetic pleasures, 
but rather as an illustration of a psychological drive 
that lays beyond the pleasure principle, which he 
calls the “death instinct” and links to repetition 
compulsion. Indeed, his elaboration of this drive 
in the book Beyond the Pleasure Principle (Freud 
1953-74b) was the impetus for his essay “On the 
Uncanny” — the latter was written between drafts 
of the former and published the year before. In the 
book, he states early on that while the enjoyment 
derived from “painful experiences” in the theater or 
art hints at that which he is addressing, they “are of 
no use for our purposes, since they presuppose the 
existence and dominance of the pleasure principle; 
they give no evidence of the operation of tendencies 
beyond the pleasure principle, that is, of tendencies 
more primitive than it and independent of 
it” (Freud 1953-74a: 17). Uncanniness is, 
for Freud, marked by a return of repressed FIG.2 
infantile complexes or amputated aspects REALDOLL 


of self that are buried in the unconscious. 


The compulsive nature of such experiences, 

which Freud suggests override the pleasure principle, is better represented by the imagi- 
nary Sandman, who inspires revulsion and fear in Nathanael in every form in which he is 
repeated, than by Olympia, whose mechanical movements, however much they hint at the 
“death instinct” lurking beneath Eros, are marked by a vacillation between life and death, 
beauty and its shadow, which is experienced by Nathanael (and the reader/viewer) as both 


compelling and strangely pleasurable. 
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In short: Freud is interested in that which drives the fictional Nathanael mad, discussing 
his case as he might that of an analysand, while Jentsch is more interested in the psycho- 
logical experience of the audience watching at a remove Nathanael’s interactions with the 
mechanical Olympia. Taken together, however, they demonstrate the complex entangle- 
ment of fantasy, reality, and psychology in considerations of the uncanny. Thinking through 
such entanglements became important in the course of making my documentary, as I jour- 
neyed into a world where all three were at play, and where I experienced repeatedly uncanny 
bodies that were horrifying for some, desirable for others and, for still others, fascinating in 
the ambivalences they produced. Indeed, one gets a sense of all three in relation to the doll 
owners in my documentary. 

All of the sexdoll owners I interviewed spoke about their doll(s) in highly idealistic 
terms, as if their dream girl(s) had become a reality. This idealization is encouraged in the 
ordering process: most dolls are ordered piecemeal (head type, body type, hair color, etc.) 
so that they are constructed to the exact contours of their owners’ desires. Such imagineer- 
ing —although directed at a precise material outcome—also encourages generative fanta- 


sies that get played out even before the dolls arrive, and that are sustained long after. 
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There is, however, for anyone experiencing the dolls at close range, a notable discordance 
between their owners’ descriptions and the stark reality—the dead weight, immobility, 
and vacant stares; the degraded silicone around the eyes, mouths, and joints of older dolls; 
hair and fingernails that had been rendered askew through repetitive use. The exterior 
reality— however obsessively tailored —seems to fade from view as the dolls become emo- 
tionally and libidinally cathected. While such relations generally happen in private, once 
made public, the contrast between the subjective experience of the dolls as desirable and 
their objective undesirability for all others can become a source of curiosity, pathos, horror, 
and even humor, depending on how it is framed. 

E.T.A. Hoffmann was far ahead of his time in contemplating the inherent drama of 
the subjective/objective split in human/non-human interactions. He was writing at a time 
when human automata were touring Europe, and his stories involving mechanical humans 
were most certainly inspired by his first-hand impressions, which were by all accounts not 
favorable. He rightly intuited, however, the dramatic and critical possibilities in stories 


of artificial love. In “The Sandman,” the awkward encounters in which the enamored 


Nathanael attempts to woo the stiff and 
taciturn Olympia to the horror of his 
friends and the intrigue of the story’s 
readers, anticipates similar scenarios in 
films ranging from Metropolis (1927) to 
The Stepford Wives (1975; remade 2004) 
to Fellini’s Casanova (1976) to Lars and 
the Real Girl (2007). Within these works, 
scenarios that might otherwise play out 
as a typical erotic or romantic encoun- 
ter are defamiliarized by the insertion 
of an artificial body. In the process, 
they draw attention to the constructed- 
ness—indeed, to the cultural and social 
programming — involved in acts of love, 
sex, and romance, raising intriguing 
questions about human agency. 

Such moments of derealization 
are not only thought provoking and 
entertaining, but also potentially erotic, 
as I discovered after meeting a subcul- 
ture of robot fetishists. I have written at 
length elsewhere about this fetish sub- 
culture, whose members call it A.S.F.R. 
(alt.sex.fetish.robots) after the internet 
newsgroup where they originally con- 
gregated online (see de Fren 2009). 
What is worth reiterating here 
is their unabashed attraction for 
uncanny scenarios. As an exam- FIG.4 | pe 
ple, although most ASFRians DETAILS OF REALDOLL 
(as they call themselves) love the 
film, The Stepford Wives, their 


interests reside less in the idea 


of the perfect robotic housewife 

than in those uncanny scenes in which the wives malfunction or get caught in a repeat loop, 
scenes beneath which forboding music plays and that are intended to evoke horror. These are 
moments of vertiginous rupture that not only offer a glimpse of the robotic programming 
beneath the ideal exterior of the Wives, but that also throw into relief the cultural norms 
through which such ideals are constructed. Indeed, in the film, such scenes serve as feminist 
commentary on the extent to which real women (and men) have been socially programmed. 
Such moments of robotic unmasking and defamiliarization recall, of course, the climactic 
scene in The Sandman, when Olympia’s eyes are removed and she is revealed as a mechanical 
doll to Nathanael. It is just such scenes that are uploaded and played repeatedly on their 
websites by ASFRians for their own erotic pleasure. 

Although human-looking dolls and robots are, since the time I made my documen- 
tary, increasingly in development and the news, they remain liminal figures existing some- 
where between the human and non-human, fantasy and reality, subjective and objective 
experience. The extent to which such humanoid creations produce uncanny effects is often 
dependent on where they fall along these continuums, but also their creators’ intentions. 
Rather than approach the Uncanny Valley as a hurdle to be avoided at all costs, it would 
serve us well to amble its many footpaths in curious exploration of its aesthetic and critical 


possibilities. 


REFERENCES 


De Fren, Allison (2009), Techno- 
fetishism and the Uncanny Desires 
of A.S.F.R. (alt.sex.fetish.robots). 
Science Fiction Studies, Volume 36, 
p. 404-440. 

De Fren, Allison (2008), The Exqui- 
site Corpse: Disarticulations of the 
Artificial Female, 2008: 
http://digitallibrary.usc.edu/cdm/ref/ 
collection/p15799coll127/id/123923 
(02.02.2022) 

— 

Dorfman, Elena (2005), Still Lovers. 
New York, Channel Photographics. 
Freud, Sigmund (1953-74a), The 
Uncanny, in: The Standard Edition of 
the Complete Psychological Works of 
Sigmund Freud, Volume XVII, trans- 
lated by James Strachey. London, 

p. 219-252. 

— 

Freud, Sigmund (1953-74b), Beyond 
the Pleasure Principle. 1920. Trans. 
James Strachey. New York. 

— 

Hoffmann, E.T.A. (1967), The Sand- 
man, The Best Tales of Hoffmann. Ed. 
E.F. Bleiler, New York, p. 183-214. 
Jentsch, Ernst (1906), On the 
Psychology of the Uncanny. 

Trans. Roy Sellars. Angelaki 2.1, 
1996, p. 7-16. 

McLuhan, Marshall (1951), 

The Mechanical Bride: Folklore of 
Industrial Man. New York. 

Mori, Masahiro (1970), Bukimi no tani 
(The uncanny valley), translated by 
K. F. MacDorman & T. Minato. 
Energy, 7(4), p. 33-35. 

Nelson, Victoria (2001), The Secret 
Life of Puppets. Cambridge Mass. 
— 

Wood, Gaby (2002), Edison’s Eve: 

A Magical History of the Quest for 
Mechanical Life. New York. 


169 


170 


THE MECHANICAL 
BRIDE. 
REALDOLLS AS 
COMPANIONS 
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Allison de Fren, the film’s director, 
is Associate Professor for Media 
Arts and Culture at Occidental 
College in Los Angeles Her 
research focuses on gender and 
technology as well as bodies and 
technology, in particular synthetic 
bodies. Even her dissertation was 
on the representation of artificial 
women since the Renaissance, see 
de Fren, Exquisite Corpse. She is 
currently a media practitioner and 
scholar. 


The Mechanical Bride is the title of a 2012 documentary about the use of sex dolls, more accu- 
rately of a particularly lifelike brand called RealDolls.' The dolls’ designers and sellers also 
have their say and there is some discussion on the development of sex robots. 

The documentary takes a clear position on the subject, as indicated by the title, taken 
from Marshall McLuhan’s book, The Mechanical Bride: Folklore of Industrial Man (1951). 
McLuhan explored the link between technology and sexuality as well as the work put in by 
women to achieve a perfect body. 

For the filmmakers, RealDolls are squarely in the tradition of the Pygmalion myth, 
the male desire since Antiquity to create and own a woman made after his own ideal. The 
accessibility of the dolls is touched upon, as well as the close emotional tie some men have to 
their dolls and the violence acted out upon the dolls that raises the ethical question of whether 
this is still violence against property or, because of their naturalistic design, something else. 
Particularly irritating are the moments that show men living emotional, intimate lives 
together with their RealDolls or discussing hate-filled aggressive violence against them. 

‘The starting point of the film was a photographic study by Elena Dorfman and 
Elizabeth Alexandre compiled in the book Szil Lovers. Published in 2005, it was the culmi- 
nation of a documentary project started in 1999, shortly after the release of RealDolls in the 
mid-1990s. The photographs were exhibited in New York, Chicago, and elsewhere (Smith 
2017: 227). Often, they portray intimate situations and emotional connection, for example 
RealDolls and their owners in day-to-day situations such as watching TV or sitting in the 
garden or at the dinner table. 

Inspired by this book and by her previous job as a digital interaction designer, in 
The Mechanical Bride, de Fren delved into the relationships between the men and their Real- 
Dolls. At the time of production, she had already documented technofetishism, also a result 
of working with almost only male roboticists in a “future technology” think tank in Silicon 


Valley (De Fren 2009). 


The Mechanical Bride is not the only documentary on the subject of RealDolls and 8111 
Lovers is not the only photographic documentation thereof. Sandra Hoyn’s photo project 
Jenny’s Soul documents a man’s intimate relationship with his doll,’ and the 2006 BBC 
documentary Guys and Dolls explores the same subject, to name just two examples.’ 
Starting from the photo book Szi// Lovers and the documentary The Mechanical Bride 
this essay first explores the question of emotional connection to RealDolls. This is followed 


by an analysis of the book, and one photo in particular, from a feminist perspective. 


“HOW DOES ONE LOVE SOME- 
THING THAT IS NOT ALIVE?”* 


Davecat is one of the documentary’s main protagonists. He lives together with a RealDoll 
he has named Sh-Chan that he considers his “companion.” He buys her clothes, make-up 
and jewelry and takes her to the park and to restaurants. He carries her picture in his wallet 
and his few friends know about and accept Sh-Chan. Davecat tells the filmmakers that he 
could imagine having a relationship with a “real” woman, but only if she accepted Sh-Chan. 
His ideal would be a ménage 4 trois with two people, a woman and a man, and a RealDoll. 
Another interviewee, an old, lonely, and sick man who dies during the making of the film, 
bought a doll that looked similar to his wife after she passed away. He lives with the doll 
in his small apartment and says that it makes him feel less lonely. Old and sick as he is, 
he doesn’t believe he could subject another person to being with him. Another interviewee 
sees RealDolls purely as a sex object with which one can do things that are socially inap- 
propriate and, more importantly, not acceptable to women. Eager not to be misunderstood, 
he stresses that he has nothing against feminism, but it’s difficult because the male instinct 
is to control women. He even repeats “control them” twice to give it more weight. Another 
man talked about his violent against the dolls. He pulls out their limbs and, one could say, 
massacres them. 

The film portrays a range of emotional responses to RealDolls, from intimate loving 
relationships in which the dolls become a partner to attempts to combat loneliness or com- 
pensate for loss to fantasies of control and power and excesses of violence and aggression. 
One could pathologize these cases and write them off as a marginal social phenomenon or as 
relationship problems of socially outcast, emotionally disturbed or lonely people. However 
the documentary asks a fundamental question about the relationship between humans and 
machines that, in light of developments in robotics and artificial intelligence, is extremely 
relevant and cannot be dismissed as a peripheral problem of a technified society: “How does 


one love something that is not alive?” This question is examined below from a historical 


perspective. En 
2 https://www.sandrahoyn.de/ 
portfolio/jennys-soul/ 
(13.06.2019). 
3 See https://www.youtube.com/ 
9 watch?v=pxCkULUnVHO 


(13.06.2019]. 


AND NEW RELATIONSHIP = u 


in the documentary, The Mechani- 


M E l ? cal Bride. Smith (2017: 227ff.) 
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5 See for example Danaher, 
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Sex robots have recently sparked a heated debate.* David Levy’s (2017) provocative state Mer OOE) Aa 


ment that love relationships with sex robots will be commonplace by 2050 is often cited. particular the Campaign Against 
Fi FR: r 5 Sex Robots: https://campaign- 
Interest in the topic is quickly confirmed by a look at media and the book market. Even the againstsexrobots.org/author/ 


robotcampaign/ (15.06.2019) 
especially Kathleen Richardsen’s 
hi-tech sex toys, and sex robots (Blum 2017). feminist critique. 


journal of the Bundeswehr, the German military, recently published an article on sex dolls, 
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Machines that interact with 
humans and are capable of emo- 
tions have often been the subject 
of science fiction in both literature 
and film. The arc spans from RUR 
in the 1920s to Spielberg’s movie 
Al in the 1980s to more recent 
movies such as Ex Machina or Her. 
There have also been many less 
successful movies that explore 
the subject, such as Electric 
Dreams by Steve Barron in 1984. 


Historically, the question must be asked whether this is a new kind of human-machine rela- 
tion or even a new relationship model altogether in which technological objects are elevated 
to relationship status. The men interviewed for The Mechanical Bride repeatedly stress that 
for them these dolls are partners or companions, a term also used by roboticists in discus- 
sions on the development of social robots. The film therefore poses the question of living 
together with artificial companions and hence a fundamental question about relationships 
to artificial Others, whereby the concept of the Other must be fundamentally questioned 
(Gunkel, Marcondes, Mersch 2016). 

Historical research has not taken this question into consideration. The few studies 
that have been conducted on sex dolls are in the main interested in the cultural history of 
the tradition of the Pygmalion myth and so in the long continuity of the desire to create and 
own a perfect woman crafted after one’s ideal. They reference the virtual creation of artificial 
women in cultural history; in literature and science fiction as well as in Kokoschka’s images 
of Alma Mahler as a doll or in Hans Bellmar’s erotic dolls (Wennerscheid 2019, Ferguson 
2010, Smith 2013). 

Historical reconstructions of the history of sex dolls also stress their function as a 
replacement for a missing human partner. The most plastic example is the “dame de voyage,” 
considered to be the first sex doll, a woman made of old clothes meant to stand in for women 
on long sea voyages and ensure sexual hygiene (Ferguson 2010: 16f.). 

With the emergence of the sex industry in the late nineteenth century, advertise- 
ments arose for sex dolls that clearly show that the key idea was to replace human but 
inaccessible female partners. With the advent of a sex industry, these puppets, as Marquard 
Smith (2013: 183) contends, must be seen in the context of mass production, commerce, 
and, most of all, consumption. This underlines the fact that these dolls are objects. In the 
late nineteenth century, sex dolls were advertised as a “perfect illusion of reality.” (Ibid.: 185) 

A look into history so reveals two connected lines of continuity, both of which are 
rooted in concepts of accessibility and objectness: the idea of the creation of the perfect 
woman made after one’s own ideal and always at one’s disposal, as well as technological 
replacements for inaccessible women. In the past, sex dolls were difficult to come buy. They 
could be bought in bordellos or porn theaters, later also in sex shops (ibid.: 191). Since the 
1970s and ‘80s, blow-up dolls have been readily available. But even if sex dolls and artificial 
women have been a pop-culture topos since the 1970s, they still were considered “dirty.”® 
It would have been unthinkable in the 1970s or 1980s to take one into a restaurant as the 
protagonist of The Mechanical Bride does. 

RealDolls have been on the market since the 1990s, and the number of almost 
exclusively male customers has steadily risen (Ferguson 2010: 40f.). As Anthony Ferguson 
observed, “suddenly the sex doll was the in thing.” (Ibid.: 3) On the one hand, this is a 
result of technological progress, with the development of new materials and their increased 
availability and with the increased ease of production of lifelike dolls. A second factor influ- 
encing this development is the internet, where aficionados of sex dolls come together and 
bolster one another in their attraction to these objects. A group of people arose who asserted 
themselves as “technosexuals.” (Ibid.: 4) Finally, the web made the dolls more available 
and easier to buy. But there have also been societal changes that play an important role in 
the spread of sex dolls, such as a public discussion of sexuality and a greater acceptance of 
previously deviant forms of sexuality. Sherry Turkle has even been criticized for her critical 
positions on sex dolls and sex robots: “I was asked if my opposition to people marrying 
robots didn’t put me in the same camp as those who oppose the marriage of lesbians or gay 
men.” (Turkle 2010: 6) The journalist who interviewed her called this “species chauvinism.” 
(Ibid.) On the one hand, this is in line with posthuman idea, but is also exhibits, as Turkle 
comments, an “openness to seeing computational objects as other minds.” (Ibid.: 5) 

Historical research on the history of sex dolls has not explored the possibility of 
intimate emotional relationships to them. It is important to note that it is only in recent 


decades that, due to the developments described above, there has been any debate at all 


about RealDolls and the like as “artificial companions.” Are sex dolls, as historical literature 
in the main contends, still primarily sex toys, surrogates for human partners or masturbation 
aids? Or are they, as in the documentary under discussion here, also companions and part- 
ners? Are machines being given a new emotional and social role in our societies? 

There has long been research on emotional bonding to objects. But these have focused 
on the symbolic ascriptions given objects and their emotional functions, not examined things 
as partners. Many of the questions asked in research on material culture are also pertinent 
to digital objects such as social robots and lifelike sex dolls. At the same time, RealDolls 
and sex robots seem to have a status of their own. Recent theories such as agential realism 
and posthumanism strongly proclaim the obsolescence of classic lines of demarcation and 
of the dichotomies drawn in the human-object relationships. They declaim anthropocentric 
thought, demanding that forms of relationship should not be limited to human-human or 
human-animal. However it should be asked whether these normative theories provide an 
adequate description, or whether relationships are more complex and cannot simply be cap- 
tured by the dichotomy constructed here between dual, demarcated categories on the one 
side and decentral, hybrid categories on the other. 

Historically, the 1990s seem to mark a caesura where types and models of relation- 
ships diversified and all forms of relationships abounded: dual, mixed, hierarchical, equal, 
homogenous, and heterogeneous. This is the time in which RealDolls entered the mar- 
ket and Tamagotchis became accepted as digital pets, followed by Furby and Aibo. Today, 
robots stand behind hotel reception desks, and are planned to talk to residents of seniors’ 
homes or provide information in shopping malls. Digital objects and deceptively lifelike 
dolls began to populate our social worlds in the 1990s, and new forms of relationships 
developed. 

Neither RealDolls nor Tamagotchi ever became mass phenomena. ‘They did not 
become widespread substitutes for human partners or living pets. Yet what did develop, it 
is safe to conclude, is a new relationship to machines that has gained legitimacy as an emo- 
tional bond. Sherry Turkle (2011: Part One) focuses on this caesura when she speaks of the 
“robotic moment.” She has observed that with the development of social robots computers 
went from being an “instrument” to which one could be emotionally connected to becom- 
ing “companions.” (Ibid.: 3-4) While discussions in the 1980s on whether computers can 
“really” think still saw them as tools and stressed their artificiality, there has been a massive 
shift in thinking (ibid.). For many people, it has become natural to accept emotional rela- 


tionships to robots. 


HISTORICAL 
DIFFERENTIATION: 
CONTINUITIES, 
DISCONTINUITIES, SHIFTS, 
AND INNOVATIONS 


This must be narrowed down historically and also differentiated. When digital objects or 
so-called social robots are touted as companions or partners, then human-machine relations 
and their social role has changed. Yet this is not a “from-to” development in which instru- 
mentalization is supplanted by partnership. 

Rather we can observe continuities, shifts, and the emergence of new types of rela- 
tionships, as well as a mix of all of the above in emotional relationships both between 


humans and machines as well as between humans. 
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Bloch, The Sexual Life of Our Time, 
cited in Ferguson (2010: 17). 
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There have to date been very few 
empirical surveys on how people 
deal with RealDolls or the first sex 
robots, but the few that exist are 
illuminating. See here for example 
Ferguson’s heuristic survey in a 
doll forum (2010: 114-120), 

and Scheutz and Arnold (2018: 
247-260). The former is a survey 
of users, the latter of attitudes 
towards and appraisal of sex 
robots among the general public. 


Ke) 


Turkle (2011) gives many 
examples that suggest that 
cognitive knowledge does not 
affect emotionality. 


Continuities and the emergence of new forms exist side by side. Particularly in the context 
of sex robots, some are still asking the question of whether it can be “real” love and “real” 
emotions, while others like Davecat, whose “love life” is shown in The Mechanical Bride, are 
already living that model with conviction. 

ELIZA is another prominent example of continuities. As early as the 1960s, Joseph 
Weizenbaum (1978) was disturbed when his secretary began to communicate intensively 
with the program he had written to simulate a psychiatrist, even sending him out of the 
room to guarantee her privacy. That people enjoy communicating with machines and trust 
them, or are open to a new form of intimacy with machines was already made obvious by the 
example of ELIZA. And it has been a topic in pop culture and science fiction even longer. 

We can also see clear continuities in the figure of accessibility. The “benefits” of sex 
dolls, their accessibility and compliance, was already touted in the early twentieth century as 
an advantage over human women. Dolls never become jealous or start a fight and: “they are 
always ready, always compliant.”’ Advertisements for sex dolls underlined this tractability, 
as well as the advantage of avoiding tiring or unpleasant social situations. Similar claims are 
made for twenty-first century sex dolls: RealDolls claim to be perfect for men who are shy, 
anti-social, or disabled. Furthermore, they do not need to be entertained and never get sick 
(Ferguson 2010: 47). David Levy (2007: 211) sees another central benefit of sex robots; they 
make it possible to avoid the “constraints” and complications of “more conventional sexual 
relationships.” Robots also guarantee to be loyal partners who never stop loving “their” per- 
son (ibid.: 22). In the early twentieth century, owners could choose from different models, in 
the meantime, these dolls can be built to suit their owner’s tastes. RealDolls and sex robots 
can be put together from modules with an array of body types, eyes, mouths, etc. 

A further continuity is the ability to act out negative feelings. ‘The entire palette of 
human emotions is mirrored in how people interact with sex dolls, as outlined in the dis- 
cussion of The Mechanical Bride above. Sex dolls are tortured and mistreated. Newly opened 
robot bordellos report that their customers’ lack of restraint is a bottomless pit of socially 
unaccepted practices. Recently, people have even developed “cathartic objects”; robots made 
for venting steam (Moorstedt 2019). 

How people interact with the new digital Other all too clearly mirrors the abreaction 
and bolstering of emotional behaviors, desires, and ideals that have long existed, but now 
are intensified in the context of technologization. They can be acted on unimpeded and are 
partly also accepted. The digital Other should thus be seen in a long continuity of human 
emotions. It is necessary to pose more nuanced questions about which emotional behaviors 
is strengthened through the use of digital objects and which are not, for example tolerance 
of non-compliance or of the idiosyncrasies of the Other. 

Descriptions by the owners of RealDolls themselves reveal interesting insights 
into questions of change, the novelty of relationship models, and the status of RealDolls.® 
Almost all of them say that their relationship to the doll is emotional. Some of them report 
that this emotionality developed over the course of time because they had to take care of and 
repair the dolls and because they got used to them being around. They gave them names. 
Some said they would be very lonely without the dolls; others explained, that they were glad 
to have a silent doll around after an exhausting and communicative day at work. One man 
said the doll owned him, not the other way around. It should however be stressed that doll 
owners, asked their opinion on David Levy’s prognosis that humans would be falling in love 
with robots by 2050, answered that this was neither possible nor desirable. They saw a clear 
division between loving a human being and the emotion they felt for their dolls, and never 
forgot that they were dolls (Ferguson 2010: 114-120). 

That makes it clear that relationship to digital or artificial Others must be consid- 
ered as unlike human-human relationships, but nevertheless as a new form of relation- 
ship. It should however be asked whether this is as easy to differentiate as two distinct 
types of emotion on the affective levels as it is on the cognitive level.’ For recent studies 


make it abundantly clear that humans are quick to anthropomorphize robots. In one vivid 


example from 2014, two Canadian scientists developed a robot, Hitchbot, to test whether 
robots could trust humans. Hitchbot grew a devoted fan base, that treated him like a friend. 
The bot could hold simple conversations, take photos, and post on Facebook and Twitter. 
It hitched first through Canada and then through Germany. Hitchbot could not move 
autonomously, but only with the help of humans, and obliging humans who offered a lift 
were always found. But one day, on a further tour through the USA, Hitchbot was irrepara- 
bly damaged by “vandals,” as the press scandalized. However, such anthropomorphizing of 
technology is not a new historical phenomenon. 

What is new, is this type of collective empathy with a robot and the broad interest 
of the press. Certainly the technical development of seemingly being able to “answer” plays 
a role, or the interaction with robots that is perceived as dialogical, because they react, look 
at their human partners and even sometimes recognize them, quasi relating to them. The 
above is also true for sex robots in contrast to conventional sex dolls. But even the RealDolls 
of the 1990s possessed a new quality of ostensible “realness” that changed the emotions of 
their users due to their weight (usually around fifty kilos) and their lifelike “skin.” (Smith 
2013: 241) 

It cannot be overseen that new relationship models are cur- 
rently developing that we must put into historical, cultural, ethical, 
and social perspective. Technology challenges existing concepts of 
relationships—to date thought of as being reciprocal—expanding 
and shifting them. RealDolls and sex robots, as well as the many 
other so-called social robots, force us to reexamine the question of 
what emotions are. These are complex questions. How should we 
characterize the emotions described? Are these historically novel 
feelings, or are they comparable to love for humans or animals? Is 
there a difference between cognitive differentiation and emotional 
connection? While human and machine thinking can be clearly dif- 
ferentiated, it is much more complicated to distinguish between the 
often attested “real” feelings for humans and simulated feelings for 


computers and robots. 


FIG. 1 


STILL LOVERS? DORFMAN, WEDDING PHOTO 


The gaze is drawn to the image framed by the open door. Behind the opening, somewhat 
blurred, stands a man in a suit, a groom who is holding a woman in a wedding dress in 
his arms, his bride (Dorfman 2005: 17). It looks like he is getting ready to carry her over 
the threshold that marks a transition for both of them from one life to the next. In so 
doing, following a further meaning of this well-known and widespread patriarchal wedding 
custom, they leave evil spirits behind when entering the new world, a house protected by the 
man. However, the man in the photo is not holding a woman in his arms, his bride is not 
living, but mechanical—a synthetic, expressly non-human but rather technological Other. 
He is holding a doll in his arms. This is no ordinary doll, it is not a toy, or a tortoise collect- 
ible or a Barbie doll, but a life-size sex doll formed after a female ideal. 

The photo is taken from a book entitled Szi Lovers, the photos were taken during 
and influenced by a documentary shoot by Allison de Fren. The American photographer 
Elena Dorfman travelled the United States in and before 2005 with the French psychologist 
and journalist Elizabeth Alexandre visiting the mostly male owners of sex dolls, observing 
their daily lives, interviewing them, and photographing them together with their dolls. 
The first word of the title, Szi, refers to both the fact that the Lovers have been caught in 
a still life, but also to the silence in the photo of the newlyweds, the pose they have taken, 


the woman in the arms of the man, frozen before crossing the threshold, their expressions 
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immovable. Are they still or silent lovers or are they still lovers now that they are married? 
The photo is the first of a series in the book that takes us over the threshold into the home of 
the supposed newlyweds in three further photos. ‘The artificial bride is also photographed in 
profile in her wedding dress and wearing sunglasses. In the next photo, the bride and groom 
are shown hand in hand, their entwined hands are the focus of the image. In the fourth 
photo, this opening series ends on the bed. We see the doll lying there in her dress on her 
back, her hands are entwined, her left leg is falling of the bed, her left foot peeks out from 
behind the white hem. 

The characteristic quality of these and the following images crystallized. Their fram- 
ing and arrangement gives viewers a look into scenes that are familiar from our daily lives, 
but each is composed, given particular emphasis through Dorfman’s use of color. Her stag- 
ings make it easy to see what otherwise is not visible at first glance. For the male owners, the 
dolls take the place of a woman that they believe belongs to them. She is therefore carried 
over the threshold, placed on a chair, and lain on the bed. She is prepared for these events; 
she is dressed, her make up and hair are done, and she is positioned accordingly. Dorfman 
follows each of these procedures, helps to create them, and presses the shutter release. In 
her preface, Dorfman tells about meeting the doll’s groom. She got to know Peter through 
internet research and asked him whether he would be willing to be photographed together 
with his doll. Peter agreed and showed Dorfman his home. He presented his doll, to whom 
he had given the name “Azra” and whom he wanted to quickly marry before his real fiancée 
moved in and he then sold the doll as she had requested. His real fiancé, Peter said, did not 
mind that he owned and had sex with a doll, but she did not like that the doll would get 
neither fat nor old (Dorfman 2005: 4-5). 

Dorfman expands on this first contact and first conversation to describe the meetings 
with doll owners that followed. From the moment the dolls were delivered and taken out of 
the packaging, they became foils for their owners’ imaginations and were thus brought to life 
by them. The dolls were given names, biographies, and individual characters: Ginger-Brook, 
a lonely Californian woman who wants nothing to do with her family; CJ, an anchorwoman 
who reported the news every evening; Sidore, a troublemaking boarding school student who 
grew up in the suburbs of Tokyo and is now a Goth. Dorfman became a witness to these 
worlds, she visited them and wrote about them that they are not meant for the outside world, 
since the “relationships” that are manifested within them take place in the private sphere and 
behind closed doors. Dorfman was given a peek into what happens behind those doors, where 
ideals, phantasies, and desires are acted out. They are images of men dressing their dolls, 
caressing their breasts, reaching between their legs from behind in bed or ostensibly stimu- 
lating them while they sit in a chair. They are images of men living out their fantasies, ideals, 
and desires with one or more dolls, with or without the consent of their human partner. 

When we look at Dorfman’s stills or de Fren’s moving pictures, are we, as the title of 
the book and the images selected suggest, looking at true lovers, even spouses? Can we, as 
Dorfman advances, speak of relationships that have developed between doll owner and doll? 
(Ibid.) The answer is no, since this is not love for another person. This is not about desire for 
a physical partner, through which a relationship can develop that feeds off consent and non- 
compliance and is characterized by mutual empathy, care, compassion, challenge, respect 
and critique and maybe culminates in marriage. It is also not about social life, where ques- 
tions about the other person arise, about their otherness, their a/terity, which is unavoidable 
in intimate relationships or in work relationships (Bedorf 2011). Others get in the way, they 
can hinder me or support me. I can try to get close to them or avoid them and I am respon- 
sible for them, as they are for me. It is not about any kind of being together that was chosen 
through mutual respect and that is constantly renegotiated; a companionship in which the 
Other is wife, lover, friend, companion, and partner. Rather, the aim is to eliminate any 
chance of mutual togetherness and to focus male desire only on that that does not, or better 
cannot, come from an Other, for dolls do not speak or act. They offer no resistance, because 


they are not human. 


It is here that a feminist perspective begins, for it is clear that men can own these dolls 
and dominate them. Crossing the threshold with the doll in arm is therefore a movement 
towards ownership. It is an appropriating, controlling, and also violent movement, bol- 
stered by the history of this custom. Its origins lie long before the banning of demons in the 
so-called Abduction of the Sabine Women shortly after the foundation of the state of Rom. 
Under Romulus, the Romans lured their neighbors into their city, inhabited predominantly 
by men, and then dragged the women over their thresholds and raped them to secure the 
future of their community. 

The image discussed above is followed by many other images with other thresholds 
and transitions that make visible continuities in relations between the sexes. This is mirrored 
in the relationship to technology, also in the sense that no matter what the company or 
buyer says about sex dolls being harmless, or even fulfilling a social, therapeutic function, 
they continue to support beliefs about what men are due and what women are for (Murphy 
2017). ‘They underline that sex is something that men get from or do to women. Sex is not 
something two people can enjoy. It does not demand care, compassion, respect, or empa- 
thy. Accordingly, advocates of sex dolls describe “companionship” as a one way street for 
men—challenges, noncompliance, feelings, and ideas are not valuable or desirable qualities 
in a woman. 

In this context, being together is solely about fulfilling men’s desires. That is what the 
dolls are made for and they should be looked at from that perspective, for it is what moti- 
vated the creation process, which Alexandre developed in her essay for Dorfman’s catalog, 
together with the men they visited. Creation begins with Adam who made Eve as a female 
“prototype” of his desire, as a “gynoid” or “fembot.” But Eve is human and not synthetic, like 
Galatea, Pygmalion’s ivory statue, modeled after a woman (Dorfman 2005: 9-11). Synthetic 
women offer men the perfect solution to feminist movements. It you won't submit to us, we 


will create women that will (Smith 2017). The ideal women is therefore, we can conclude, as 


always, not human (Murphy 2017). 


Originally published in German: “The Mechanical Bride. RealDolls als companion und Objekt der Verfiigbarkeit”, in: Martina 
Heßler (Hg.), Technikemotionen, Paderborn: Ferdinand Schöningh, Reihe Geschichte der technischen Kultur, 2020, 369-384. 
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PROMETHEAN SHAME? 
PANDORA REFRAMES: 
ELENA DORFMAN’S 
ORIGIN OF THE NEW 
WORLD 

OR, DIGITAL ARTS AND 
MATERIAL CRITICISM 
IN THE BREACH 


JAMES TOBIAS 


The digital prints line up across a wall, marching 


into and out of a corner of a common room in an arts 


university where a symposium has been taking place. 


~ _ 
RS À ] The “V” shape of the march of images, in their seri- 
aad ated projection across the now extended corner of the 
‚ill T E 2 room, suggests the perspectival vanishing point with- 
» in each photographic scene whereby a triangle 


» 


repeats: at the apex, above, a single breast; one thigh 


forms the lower left corner as it exits the frame; 


across the pudenda the third corner is formed by a 


knee at the lower right of the frame. Within each 
triangulated scene repeats an assemblage of silicon 
body parts— specifically, 
in the foreground, pudenda 


FIG.1 decorated with more or less 


ELENA DORFMAN, ORIGIN OF THE NEW WORLD, füzzy clouds of hair, and 
INSTALLATION VIEW, ZURICH UNIVERSITY OF mene amar 
THE ARTS, DECEMBER 2018 meee 


or penis modeled within 


each image, the variously 
sized single-breast reclin- 
ing away from the viewer. Elena Dorfman’s series Origin of the New World remodels and 
remakes Courbet’s 1865 Origin of the World as a parade of factory-made, interchangeable sil- 
icone parts, artfully applied lighting and makeup, and digital framing and printing attached 


simply, but also precisely, to two walls of the symposium space. One reaction I noticed was 
to feel vaguely aroused—you may choose the gender of your interest to fixate upon, but 
nevertheless, you don’t commit, your eye wanders, one can’t decide— but also neutral and 
neutralized at the same time, in a kind of grating tension, as if one had taken beta-blockers 
in order to enter a sex shop (one that happens to be particularly and unusually attentive to 
product spacing, print quality, and visual and spatial presentation) in order not to have to 
feel forced to react to the provocations of the products displayed there. Your reflexive posi- 
tioning in the materialized, metered corner as it folds out into the symposium room — that’s 
the reaction, it precedes you, and it follows you, and you know it. This is your work space. 

On the far right of the right wall where the prints continue their march, and anchor- 
ing this complex assemblage of scene, space, and positioning, is a curtain. Behind this 
curtain, a lightbox is hung—as if, here, enshrouded and reserved for a brief moment of 
secret pleasure, we find the origin of this second coming—or perhaps its conclusion. You 
went behind the curtain, you found a mirrored surface, your own image looking back at 
you; turning on the machine’s light, your face fades to be softly superimposed over another 
of the images of silicon pudenda. The sense given is that what might have been assembled 
as a scene of conversation between the viewer and the artwork, or of supplication, between 
the adorer and the divine, instead turns into a kind of demonstrative mismatch: your eyes, 
your nose, your face, but less visible than “her” lost modesty, her exposed breast, her disap- 
peared face. 

The disappearance is forced and it is double; first, Courbet’s model (now thought to 
be dancer Constance Queniaux) is disappeared all over again, as she was in Courbet’s work; 
but then, too, so is Courbet’s rendering of Queniaux as faceless also, itself, disappeared. The 
violent double logic is made patent: your face only appears to you to the extent that hers 
disappears from “her body.” No mistake about it, whether masculine-endowed or feminine- 
endowed or trans-endowed, these are objects whose strangeness has to do with their status 
not simply as objects but as tools, tools for seeing, seeing not simply into the image or its 
framing, but also the platform of knowledge-making on which the image is displayed. ‘These 
images are instruments for viewing processes of feminization, racialization, sexing, upon 
the larger platform where the human is synthesized. So I refer to the body of the entire series 
in terms of feminization, as “her body” — but to be clear, this is a montage of bodies and 
occasions. I’m not referring, only, to female gender, or to processes of engendering. Femini- 
zation, racialization, sexing, and instrumentalization— not subjectification, not objectifica- 
tion, I think—is what Elena Dorfman’s Origin of the New World makes patent. And there 
is violence captured and contained, demonstrated, made diagnosable, in each image, in the 
projection of the series into and out of the corner, across the walls, in the pseudo-privacy 
of the lightbox and its curtained scenarization— violence in the technical reduction of the 
raced, gendered body to silicon molding, to perspectival imaging, violence in the reduction 
of desire to fetish. If the photographs themselves are staged documents, they are also per- 
suasive demonstrations, a balance of aesthetic experience and essayistic display. 

So the putatively neuter space of academic research or aesthetic exhibition is 
revealed—lit up, we might say, as if in a flash—as a staging platform for the way in which 
aesthetic research can, in the form of an exhibition, prompt, even format, the preparation and 
presentation of scholarly exploration. And the space so designed makes clear the demands 
on contemporary academics to inform our work with understandings, if not theories, of 
power and of violence, specifically in relation to processes and projections of racialization 
and engendering and sexing. But this display also reveals the violence of the very notion 
of the public under which academics labor in the belief that at some appropriate point we 
must enter into the public, take part in a forum, mediate art and public in some impossibly 
ethical fashion. Impossible, because if violence informs the formation of knowledge, ethics 
is challenged to differentiate itself from a violence ethics must deny, or simply descend into 
or reveal its own violences. This dynamic is in part what scholarly engagement on the actor 


and the avatar reveals as needing to be taken into account. 
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And indeed when the symposium ended, and most of the scholars left, and the public 
entered, another kind of violence, violence carrying violent threat, came with it. As Nadja 
Ben Khelifa, one of the scholars who took part in the “Uncanny Valley” symposium around 
the square of tables across the way from the “V” of Elena Dorfman’s witting, sensuous, 
engaged display of unwitting, unfeeling and unthinking yet technically precise pudenda, 
tells it, the violence captured and contained in Dorfman’s work found its reflexive enactment 
in what was, this time, an unfortunately demonstrative encounter with the public. 

A man entered the symposium and exhibition space, screaming in Swiss German. 
Surprised, and unaccustomed to the dialect, ben Khelifa asks what he is talking about, and 
informs him in no uncertain terms that she will not be shouted at in such a way. He insists 
that his “macho” behavior is to be allowed in this “common” space whether she likes it or 
not—and, well, here it is in a nutshell—a demonstration of the fact that the technological 
architecting of “the common” and the “shared” are premised on a properly violent disciplin- 
ing of racialization, engendering, sexing. 

Walking towards the man, ben Khelifa informs him that he is welcome to view the 
exhibition—and I wonder: perhaps he would learn to see that his own character is fore- 
seen, formatted to be demonstrated in the “V” of marching, reclining, presenting pudenda, 
although perhaps not in such realistic or naturalistic ways. But, ben Khelifa makes clear, 
he can see the images, and he’s not allowed to shout at her or intimidate her. With her 
approach, she aims to show the familiar stranger who feels strangely at home in the common 
space shared by artistic research and scholarly presentation that she won't be intimidated by 
his violent behavior or his implied violence. She thinks, though, that her voice, less confi- 
dent, may be giving her away. The man tales a step towards the series of images—as if he 
might deface them, that is, might deface their demonstration of his own violent identity 
made visible as violence in the expanded “V” of Dorfman’s cornered, silicon-molded, face- 
less and so sensate but non-sentient pudenda. Apparently his shame takes over for his rage; 
he fails, more an avatar of violent action to come than an actor of violence in the academic 
scene. Security is notified; the man, now a violent phantasm more than a violent act threat- 
ening violence, fades away into the halls of the academy of art. Ben Khelifa described this 
incident as a kind of “Taxi Driver” moment; reflecting that logic, we might see that the 
university, for the self-appointed arbiter of public discourse and public space, becomes a site 
of iniquity replacing the infested streets of New York City; that the subject to be saved by 
the pseudo-historical actor is an avatar of silicone and light; and that “the pimp” is somehow 
now supposed to be the scholar-curator guiding the public actor through the labyrinth of the 
technologically embodied present. 

Obviously, there’s more to be said about the powerful images that hung on those 
walls. The variations of their seriation deserves mention: a tiny phallus competes to replace 
a larger one at the wandering pleasure of the viewer; skin tones and volumes jostle in the 
feeling eye’s redoubt as our bewildered look travels from image to image; pubic hair can 
seem clown-like in color or oddly mowed; the single breasts forming the apex of the visual 
triangle composing each image rise and fall as they repeat from one image to the next, as 
if in a natural evolutionary variation of the logistics of securing, pouring, and polishing 
silicone rubber. Of course the strange, sad, violent macho failed in his mission and had to 
run away— where would he even begin if his task was to restore masculine pseudo-integrity 
with such an overwhelming and productive series of projections put in play between actor 
and avatar in this space? How would one failed subject and their one sad body attain to 
the larger implications of racialized, engendering, sexing bodies diagrammed in Dorfman’s 
corner of proliferating pudenda? The tiny cock makes the point: to demonstrate engen- 
dering, racializing, sexing in terms of instruments’ relations to other instruments— where 
“ubiquitous instrumentality” is a speculative thesis suggesting test conditions for thinking 
about the way we imagine “technological being” “being everywhere” —is to demonstrate 
lack of development, immaturity, failed organization, incapacity, the neoteny of the avatar 


as social actor and of the social media subject. 


But if, as Terry Winograd long ago observed, computers work almost by default as social 
actors, it’s a complex contradiction in terms to speak of a “social media subject” — this con- 
tradiction is worth unpacking. While the avatar as a site of digitally mediated performance 
may be conceived in terms of contradictions of digital selfhood, it’s important to point out 
that beyond the logistics of projecting, enacting, and archiving digital identity, avatar per- 
formance works not simply as mask but also as costume, and demonstrates a derealizing of 
embodiment as much as a logistics of managing identity. At stake in this derealization, too, 
seems to be a kind of shame at the sight of, and at the virtual touch of, a particular tech- 
nological style of synthesizing the human. This shame might be “Promethean shame,” but, 
indeed, to demonstrate Promethean shame in this fashion suggests an artful synthesis of 
aesthesis with ethical claim-making: here, artistic practice and critical praxis work together 
to counter the inculcation and indulgence of Promethean shame and the violences we can 
associate with it. 

Günther Anders argued in his 1956 chapter “On Promethean Shame” (Anders 
2016), part of his book “Die Antiquiertheit des Menschen”, that the sheer advanced quality 
of modern technological objects produced a kind of existential shame in the contemporary 


human subject: 


“When humans suffer from feelings of inferiority in the face of their machines, 
then they do so, primarily because they are forced to realize—in their attempt to 
measure up to their machines—that they are a poor quality’ raw material when it 
comes to being moulded into this or that machine part. Instead of actually being 
raw material, humans are ‘unblessed’ morphologically because they are already pre- 
given. Moreover, every pregiven form turns out to be ‘wrong’ and ‘already pressed 
into the wrong shape’, because in each instance the actual form machines require 
varies. In ‘human engineering” people attempt to ‘melt down’ this ‘misshapen form’ 
in order to retroactively source the material from which the required shape at any 
particular time can still be made” (Anders 2016: 51). 


Anders postulated “sexual shame” as the quintessential form of shame. Since shame as 
inability or failure means it is associated with negative intentionality, is beyond choice and 
freedom, is improper to the individual, more related to “the it” than to the subject proper, so 
shame has thus to do with a conflict between being a subject and being a thing. Since sex, 
for Anders, was pregiven, thought to underlie but be beyond intentionality, and connoted 
rather negative intentionality, the inferior and unblessed, for all these reasons “one’s sex is a 
pudendum, (...) something of which one is ashamed” (ibid.: 67). With Promethean shame, 
though, Anders argues, we take account of another “it,” that of the technological device. 
“We can, in a sense, picture the human — and this is now truly only an image—as clamped 
between two brackets, as if constrained by two forces that both challenge the ‘self’: on the 
one side the human is constrained by the ‘natural it’ (by the body, sex and species, and so 
forth) and on the other side by the ‘artificial’ (bureaucratic and technological) ‘it of the 
technological device’. The space left open for the ‘self’ is [in 1956] getting smaller, while the 
danger that the ‘self’ is crushed by these two colossal non-individual forces is daily increas- 
ing” (ibid.: 76). 

Some seventy years later, it is clear that both to the side of “body, sex and species” 
and also to the side of “bureaucratic and technological artifice” we must gain specificity by 
introducing “racializing,” “engendering,” and “sexing” descriptors; and that the cybernetic 
advance of Promethean shame now requires “the it” to be not simply “the machine” but also 
to be understood as involving the complex relation of machines to machines. A “ubiquitous 
instrumentality” discursively constitutes an improper and unrigorous surrogation of his- 
torical conditionality in too many accounts of avatarial action—as if networked machines 
could absorb into their calculating operations the disparate and discontinuous histories that 


produced them. Identifying and naming this false premise in this way is one way of revising 
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popular theories of Silicon Valley’s supposedly cybernetic imaginary of avatars holding our 
place as we stumble into the merchandized technological future, and of re-thinking ideas 
about “the California ideology” as a kind of neoliberal symptom. 

And this is where Dorfman’s installation of Origin of the New World not only prompts 
and engages with critical discourse, not only distills a kind of iconoclastic platforming 
responding to and cutting off at its knees that form of iconomania whereby the images 
of celebrity, of governing authority, and of ubiquitous technological instrumentality seek 
to flourish and triumph in their unblessed, misbegotten wills to power. Dorfman’s work 
says you can have it all now—all the racing, sexing, gendering you can consume — but you 
can't have any of it and remain visible to yourself as yourself. Still, her critical referencing 
of Courbet and the history of art also reminds us, you didn’t have yourself as such to begin 
with. The result is a short circuit that leads to a breach, and there’s no way out but forward 
in some non-continuous leap over the looming void. Aesthetic experience here forces critical 
engagement where the social actor and the networked avatar disappear into one another. We 


must invent another style of technological life. 


FIG. 2 
ELENA DORFMAN, ST/LL LOVERS (2001-04) 
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STILL LOVERS (2001-04), IS A SERIES OF PHOTOGRAPHS THAT FOCUSES ON THE DOMESTIC LIVES OF MEN AND 
WOMEN WHO DEVOTE THEMSELVES TO LIFE-SIZE, ANATOMICALLY REALISTIC SEX DOLLS. THE PHOTOGRAPHS 
EXPLORE THE EMOTIONAL TIES BETWEEN HUMANS AND PERFECTLY FORMED SYNTHETIC WOMEN, FORCING US 
TO EVALUATE OUR OWN NOTIONS OF LOVE AND THE VALUE OF AN OBJECT THAT HAS THE POWER TO REPLACE 
A HUMAN BEING. 


NEARLY TWENTY YEARS AFTER THESE PHOTOGRAPHS WERE FIRST PUBLISHED THE REAL DOLL FACTORY 
IS BUSIER THAN EVER TRYING TO MEET DEMAND, WITH THE UBIQUITY OF SEX DOLLS AND THEIR ADVANCE- 
MENT INTO ROBOTICS; THEIR ACCEPTANCE AND POPULARITY RISING IN A WORLD WHERE INCEL MINDSET AND 
EXTENDED ISOLATION AS A RESULT OF THE GLOBAL PANDEMIC ARE ESCALATING. 


STILL LOVERS SHED LIGHT ON A WAY OF LIFE THAT WAS CLOSETED AND MADE WHAT APPEARED DEVIANT 
MORE EMOTIONALLY ACCESSIBLE. NOW, MY INTERESTS ARE MORE INCLINED TOWARD WOMEN AND DOLLS AND 
THEIR GROWING SIMILARITIES. THE DOLLIFICATION OF WOMEN BY WAY OF ADVANCED INJECTION, SURGICAL 
ALTERATION AND CUSTOMIZATION AND A COMPULSION TOWARD UNIFORMITY OF FEATURES-WHAT LOOKS 
GOOD ON THE DOLL LOOKS GOOD ON THE WOMAN-AND VICE-VERSA. IN A RELATIVELY SHORT PERIOD OF 
TIME WHAT WAS ONCE CONSIDERED RADICAL-THE DE-AGING OF FACE AND BODY BY DECADES, MARRYING 
A HOLOGRAM, VISITING A DOLL BROTHEL ON THE WAY TO WORK-BARELY REGISTERS AS UNUSUAL AS MEDICAL, 
TECHNOLOGICAL, AND SOCIETAL NORMS CHANGE. 
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ENCOUNTERS OF 
THE UNCANNY 
KIND AT THE 
ORIGIN OF THE 
NEW WORLD 


NADJA BEN KHELIFA 


Since the age of Enlightenment, women have been systematically associated with nature, 
irrationality and the so-called “domestic sphere”, men in a binary opposition with rational- 
ity, rational knowledge and public leadership. ‘This is by no means an accident of history. 
Rather, the discourse of nature and the emerging shift of epistemological paradigms from 
mechanistic to physiologist and biologist approaches and the resulting humanistic anthro- 
pomorphism, which favoured the “human subject” of liberal humanist thought, rendered 
the hitherto relative and markedly socio-politically argued subjection of women since 
early modern times into an essential necessity, a truth, now based on women’s irreducible 
“nature”, by the middle of the 18th century. One could not have planned the construction 
and subjection of the category “female” more systematically thorough than the complemen- 
tary developments in the areas of science, culture, law and politics suggested, which all fell 
into place under the new ideology of “nature”. By contrast, the essential human subject was 
not defined by its body but by its mind, it was a rational coherent, autonomous, free agent 
(Hayles 1999: 85). 

So, the general discourse of “nature” that was supposed to be the basis for the hard 
fought equality amongst humanity, in support of realizing the anthropocentric ideal of new 
concepts such as the universal human rights, precisely became the reason for the discrimi- 
nation against and state of inequality of women due to their alleged “different” nature. The 
elevation of nature-based equality of human beings to become the universal norm, “natural- 
ized” and therefore cemented the otherness of those who were deemed inequals by nature 
as “essential”. 

William Blackstone, the English legal theorist of the mid-18th century, notoriously 
wrote that “[iln marriage, wife and husband are legally one person, and that person is the 
husband.’ Women just disappeared in all socially relevant ways after marriage [...] their very 
being [...] is suspended [...]” (Steinbach 2004: 267): they were virtually disowned, had no 
say over their children, were legally viewed as minors and had no say over their own bodies. 
The charge of rape during marriage was legally non-existent as it was considered a contra- 


diction in terms. 


A century after Blackstone’s words, at the time Gustave Courbet accepted the remittance 
work of painting “The Origin of the World” nothing had fundamentally changed about 
these conditions. If women were considered embodied nature, and giving birth their destiny, 
men’s task, according to the rationalistic nature-ideology, was to dominate nature, outward 
and inward nature, and hence, not least of all, women. Therefore, Courbet’s flaunting of 
his model’s private parts on canvas was not at all a “feminist act” nor an act of feminism in 
order to free the “femme covert”, but, on the contrary, the perpetuation of the fantasy of 
female domination by men with the means of pornographic art, commissioned, composed, 
and painted by a man, sold to several other people, one of whom again commissioned yet 
another man in order to create a cover up of the motif, in order to better control who may 
and may not see “her”. 

Today, even though the worst violations of women’s rights by local laws and regula- 


tions are to be found in non-Western states, there are gruesome exceptions, e.g. the human 


rights violations by law with regard to transgender people. 


Moreover, examples of everyday culture show 
that the origins of the nature-discourse are 
still viable in Western societies for example 
in the widespread devaluation of knowledge 
of any kind generated by women working as writers on the internet, as artists or academics. 
The abject misogyny displayed in the digital realm is particularly staggering. Here, violent 
insults and death-threats are offered as “criticism” of the work of female writers especially 
female writers of colour and trans women writers. Sexualized violent fantasies center around 
the alleged site of “female destiny”, the reproductive organs: they threaten “sterilization”, 
“cut[ting] out the womb”, and “rape”, in order “to make her stop writing”; suggestions to “go 
be a stripper or a whore or just hang yourself”, culminate in the judgement that the female 
writer is “subhuman”.' 

That the gendered identity of the creator of a work of art may play an important part 
in the public’s willingness to accept their creation is suggested at the example of the replicas 
of Courbet’s “The Origin of the World” in various media. At least statistically, creations by 
female artists seem to have been received more harshly and were cancelled more often, 
no matter the stance they took towards the original.? The continuity of constructing the 
essentialist “female destiny” by conflating woman and body in conjunction with the 
devaluation of “her” competency through violent fantasies that could very well be justified 
by the European Enlightenment’s imperative of “nature domination” by the humanist 
subject is hard to miss. 
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Patriarchal care and violence are the two sides of the same coin even in late 20th and 
21st century Western cultures. Courbet’s 19th-century work perpetuates an image of 
women according to 18th-century gender-roles into its time and further into our present. 
‘The woman depicted is an aestheticized body, manufactured and exposed in order to be 
admired and assaulted. Considering the ongoing support for the construction of binary 
gender in order to secure power for the liberal humanist subject, it becomes less surprising, 
that the critical sardonic treatment of Courbet’s subject sparked outrage: 

Elena Dorfman’s exhibition, The Origin of the New World, provided a setting that 
commented on and refuted the very logic of Courbet’s “natural order”. Overseeing the exhi- 
bition, my encounter with one visitor, who still seemed to buy into the ideology of binary 
gendered power-relations turned out to be so self-consciously performative, that I neither 
saw it coming nor thought it possible that I would be able to witness an extreme uncanny 
reaction like that—all at once to the artwork, to the absent artist, and to myself—in an 
exhibition that took place in the framework of a workshop on “The Uncanny Valley”. 

For said middle-aged, male, white, and, judging from his vernacular, Swiss visitor, 
i.e. a self-evident “human subject”, the circumstance that on this December morning in 2018 
in the exhibition room “Aktionsraum” at the Zürcher Hochschule der Künste (Switzerland), 
contemporary photographer Dorfman’s work was overseen by yet another woman, seemed 
to break the heteropatriarchal fantasy of women that he apparently and mistakenly expected 
to find on all levels of this exhibition, that featured a series of photographic replica, unfa- 
miliarizing Courbet’s painting. In a sense, the visitor made an honest mistake, considering 
that the 19th century artist’s painting reaffirms the nature-ideology and arguably violently 
objectifies the female body as the epitome of “her” gendered identity—in popular culture 
comparable, perhaps, only to Alfred Hitchcock’s fantasy in Psycho (1960) — and therefore 
must have been disappointed by Dorfman’s replicas, which re-appropriate the representation 
of gendered bodies as markedly “unessential”. 

The visit was short, even if not a moment too short, and accompanied all the way by 
an outburst of his, provoked only by the highly intended wrecking of the visitor’s gender- 
fantasy by Dorfman’s work. I was the only other person in the room, so he made an, albeit 
stupid and ultimately unsuccessful, attempt at reinstalling the “due” gendered power- 
relations in this setting. On entering the room, he took some quick steps towards the photo- 
graphs, looked at them, then yelled at me something unintelligible in Swiss-German while 
pacing agitatedly, aimlessly and too quickly through the room waving his arms uncon- 
trolled. I asked him what he said in German and he screamed, “Did you do this?”, looking 
at me and stretching out his arm like an arrow pointing in the direction of the photographs 
arranged on two adjoining walls of the room. I replied in the negative and that there was 
no need to scream at me. Then he yelled, angrily sneering at me: “You don’t seem to like 
my macho-behaviour, do you?” And whether ‘I thought that it all belonged to me’, re- 
minding me that “this is still acommon room.” Not quite sure what was going on, I told him 
that he cannot scream at me, and asked him whether he belonged to the university. 
He walked towards me, and menacingly said “What did you say? What did you say?” 
I walked towards him as well to show him I will not be intimidated, although my voice 
probably gave away that I did not feel terribly confident when I told him that he was ‘welcome 
to watch the exhibition and then leave’, but that he ‘could not speak to me any longer.’ I got 
my smartphone, and pretended to do something with it. He paused, made some jumpy steps 
towards the photographs for a moment and then, quickly and for once focussed, left the 
room. I was slowly but visibly following him out on the platform in front of the “Aktions- 
raum”, pretending to be ready to make a call, while watching where he went. He nervously 
turned around a few times while making his way further through the building. I notified 
security immediately. 

Why would looking at Dorfman’s photographs aggravate the visitor? “Did you do 
that?” His fantasy-bubble burst in the moment he approached the photographs more closely 


and was confronted with the artificiality of the replicas of the exposed female genitalia in 


a doll, marked by the visible staples. His moment of the uncanny valley, the shattering of a 
fantasy, even a betrayal turned the exhibition into a performance, whose clash of rules was 
anticipated in the laboratory setting of the exhibition, which challenged the visitor. 

The uncanny is the realization of having been betrayed. Something that from afar 
looks familiar, like a person, changes upon closer inspection into something not quite 
“genuine” or “authentically” “human” according to the observers expectations. 

This sense of having one’s expectations disappointed or even of being betrayed runs 
deeper than an optical illusion or a sensory disorientation such as the dismantling of the 
visual illusion about merely humanoid body parts that were formerly mistakenly perceived 
as “human”, guided by one’s expectations. Rather, this disappointment is a betrayal of the 
second order, a debunking of another illusion that goes with the first one— here, a betrayal of 
the human subject’s privileged expectation — that the Western cultural hierarchy of binary 
sexes and gendered power-relations is still in place, everywhere and self-evidently, namely 
the well-practiced visual and political hierarchy of the available female body and the (male) 
onlooker, as theorized in Laura Mulvey’s often cited “male gaze” and Michel Foucault’s “eye 
of power” that produces and establishes a “visual sovereignty” over the “genital body” in the 
first place (see Uparella 2018). 

Courbet’s painting demonstrated this availability in abundance, curtailing the female 
body to his liking; in the absence of a head, arms, legs below the upper thighs any form of 
agency is denied. The naturalistic depiction of torso and vulva are exposed to the penetrating 
gaze of the viewer—violence unpierced by irony. The ideological context constitutes the 
enabling rationale: the complete conflation of the female body with the social identity and 
destiny of women due to the nature-discourse in the liberal humanist tradition. 

The visitor’s disappointment with Dorfman’s photographs then, stems from their 
withdrawal from the nature-discourse. The photographs are not in line with traditional 
representations of the female body. The refusal to supply an ontological basis for the repre- 
sentation of gender builds up a resistance towards the viewer. 

The Uncanny Valley model is linked to the nature-ideology since it features both, 
an implicit, undisclosed human subject and others; without them, however, the model 
would not generate the outcome that Mori depicted as robust and therefore as the basis for 
his claim that the UV-model could help in telling apart human beings from robots. Mori 
claimed the UV-model in robotics would help to define the human being. Zhe human being 
that is envisioned by Mori to perceive the uncanny is, however, not without presuppositions. 
Considering its own unthinking assumptions about the human being, the UV-model rather 
generates the outcome it inserts in the first place, in other words: it does not merely żel] who 
is a human being but it works on the perpetual making of the human being. Mori’s findings 
indicated a binary between a healthy human being and a human being with prostheses: 
Whereas the former never generates an uncanny reaction, the latter a/ways does. It seems, 
the mechanism of the UV-model unthinkingly implies aspects of the life-world of the per- 
ceiving subject, to exclusively fit a “healthy human”: The model implies the experiences and 
social settings of a “healthy human” as default, which is constitutive for what is perceived as 
uncanny in the first place. 

Considering the model’s intersubjective setting, it is highly unlikely that someone 
with a prostheses would react “uncannily” towards the unexpected encounter with a person 
who has a prostheses as well. Inferring from this, Mori’s UV-model, informed by European 
theorists from the 18th and 19th centuries, constructed a perceiving human subject that is 
presented as “universal” but that, much like the human subject in humanism, is defined by its 
tacit presuppositions. Insofar, the UV-model is a marker of perceived personal and socially 
institutionalized alterity, that renders distinctions within the species of human being— not 
in-between the human and the technological other—on the basis of personal familiarity. 
‘Therefore, it perpetuates the presuppositions of this setting, and renders them visible, rather 
than generating new findings. This is the rendering visible of extant discriminatory distinc- 


tions between the “human subject” and “non-human people” (Morton 2017). 
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The outrage over female agency was twofold: “Did you do this?” I did not, but Dorfman 
did: she took back the representation of women from the remnants of 19th century hetero- 
patriarchy that still affect women’s lives today, and— intricately linked— she rejected the 
hegemony over the construction of gender that is utilized in order to exert power over 
those people who are irreducibly excluded from the category of human subject in liberal 
humanism. By re-appropriating Courbet’s artistic subject, female artist Elena Dorfman’s 
rendering destroyed the purely heteronormative perverse fantasy of the availability of the 
“embodied female” devoid of agency, a maimed “genital body” depicted in Courbet’s origi- 
nal, and turned it into a gender-fluid fair of, an— on closer inspection obviously— artificial 
figurative assembly, that makes light of body parts as spare parts with obvious seams and 
staples comprising an array of artificially coloured pubic weaves, as well as vulvas, breasts 
and penises in various shapes, sizes and skin-tones, which are mixed and matched like the 
sterile custom-fit industrial set-up on a website for the interior design of a car. The tradi- 
tional view of binary normative gender categories that function as essential and therefore 
stable categories of identity is undermined. Thus, any power-relations depending on the 
stability and binary of sex and gender let alone gender-relations is presented by Dorfman as 
all, irreducibly constructed, illegitimate and doomed to fail. As a result, the preconditions 
for the Enlightenment idea of the “full human subject” are delegitimized. 

Why would he refer to himself as “macho”? The then recent discussions in the wake 
of the semi-global “Me too” movement have obviously familiarized him at least super- 
ficially with the reality of issues around the objectification and abuse of as well as the dis- 
crimination against women, which he - also in line with the still mostly gendered roles of 
perpetrator and victim - has already lived up to in advance by acting as if he was personally 
charged with something (“You don’t seem to like my macho-behaviour, do you?”) and had to 
defend himself against all, the resistance of the photographs that announce the invalidity of 
the heteropatriarchal world view, against my refusal to act in accordance with my gendered 
“destiny” as a foreign woman, and against the rejection by the whole exhibition that he as 
the “human subject” is in power, including especially the power of artistic interpretation, 
of what can and cannot be exhibited: “Did you do that?” Curiously, precisely by acting 
out this “wrongfully accused male” perpetrator-fantasy in his performance, he enacted his 
abusive fantasy for real in the first place and directed it towards me. He enacted a backlash 
against his and like-minded people’s delusion that allegedly women have “taken over” (“You 
think this all belongs to you, right?”), which is reminiscent of the fear of losing those socio- 
economic and legal privileges most white men have been accustomed to for centuries at the 
expense of women’s and PoCs rights and freedoms. 

This is nothing rare either, since the cycle of inadvertent actual “machismo” while 
making the case against (or rather whining about) being unfairly held in contempt for the 
sins of a few other (white) males who (tried to) abuse(d) their position of privilege “for real” 
is far spread. 

The encounter with an uncanny being is described by Morton (2017: 134) as some- 
thing that is in the same category with encountering an “abject”, “unclean” being. Typically, 
the aim is to get rid of it, through the exertion of violence. In patriarchal societies and 
cultures women’s bodies have traditionally been considered abject; the resulting violence 
against them was justified by the imperative of the domination of nature. “We make beings 
extermination-ready by designating them as uncanny” (ibid.). The intended speciism of 
the Uncanny Valley (humans vs technological others) needs to dehumanize people, con- 
structing them as non-human people in order to make a clearer distinction between the 
species (Morton 2017: 135) which benefits the doctrine of the hegemony of the autonomous 
liberal human subject. So, at the lowest point of the uncanny valley, we then find all kinds 
of “othered” people according to all structural discriminations we can think of: sexism and 
gendering, racism, anti-semitism and ableism. 

“You don’t own this”: this final remarkable meaning points to the core of the matter. 


How can he, a Swiss man be subject to the rules of an-other that is neither male nor Swiss? 


It is about ownership of place and hegemony of interpretation. He caught me red-handed 
while I tried to get away with something, namely changing, albeit indirectly, the order of 
how things are done. He would not let me “sneak by”. 

The uncanny is first and foremost a judgment that there is something fundamentally 
“other” that tried to pass, to sneak by the detector which singles out the uncanny. The sense 
of uncanniness marks the detection of a betrayal which speaks to an intrinsic entitlement 
of that perceiving subject. Whether a perceiver is considered a human subject themselves or 
not, is the most important precondition for sensing uncanniness in another in the first place. 
The assurance of belonging, that resides as self-evident in the human subject, is insurance 
that they are considered the human norm and that the object that sparks uncanniness may 
be not. The Uncanny Valley unthinkingly became a normative model, in place to detect 
differences from the norm. Therefore, the UV merely measures violations of a society's 
social norms, an effect that is exacerbated or diminished by the respective individual’s own 
experiences and horizons. Thus, the uncanny effect detects anything that is considered 
a deviation from the normative subject. “Disability” and “gender” are merely two of the 
triggers of the uncanny, if we consider Western culture as an indicator of who is fully con- 
sidered a human being. 

If an uncanny reaction is a reaction of extreme negative surprise and the feeling of 
betrayal towards a being that tried to “sneak by”, to “pass” as human (= healthy, human sub- 
ject), and if further the reference for what is such a full human being are the traditions of 
a societies culture and history, it is no wonder that not all genders are included in the idea 
of “human”. A woman’s body, e.g. is still considered in some fields a deficient male body, as 
exemplified by the gender data-gap in medical research. 

Against the backdrop of a heteropatriarchal attitude steeped in the ideology of a 
naturalized order, a “symbolic revulsion” (Ben-Ze’ev 2018: 207) towards everything that 
deviates from such norms, certainly against Dorfman’s photographs that invoke trans- 
sexual and intersex contexts and dismantle the normativity of binary gender, could activate 
an uncanny reaction in the respective human subject if representations of the fluidity and 
plurality of sex and gender as well as non-heteronormativity are perceived as “unnatural and 


non-human.” (ibid.). 


ee 
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FACE FRACKING 


ANTON REY 


FIG. 4 
BERTOLT BRECHT 


PHOTO: PAUL HAMANN 


“God hath given you one face and you make yourselves another.” 
Shakespeare, Hamlet 


Imagine you wake up after a bad night with little sleep, facing shortly ahead an extremely 
important video conference. The mirror tells you that this is not a winner’s look, you have 
to do something about it. 

Imagine you recall last Sunday’s face you recorded, when you were in best of moods, 
well rested and with persuading looks. A semicircle with your phone-camera around the 
happy visage was perfectly enough, and now the registered mesh fits fine for the upcoming 
meeting. Nobody will notice any fake since you are speaking in real time with not a split 
second of delay. Your opposite will never remark any difference, even with the best will in 
the world. The screened face comes in perfectly true appearance with an original voice and 


live reactions. Bright smile, perfect performance and a convincing presentation of yourself. 


Berowne: “Now fair befall your mask!” 
Rosaline: “Fair fall the face it covers!” 
Shakespeare, Love’s Labours’s Lost 


The face marks identity and variability of the same. What we project into a face is often 
our idea of what might hide behind it—or, more generally and seen from both sides of 
the surface: what we presume lies on the other side of the mask. Just as the Greek term 
“pros-opon” for face stands for the seen face, the visage open to my eyes, while the Roman 
“per-sona” means the sounding through the mask, referring to the source behind it (Belting 
2013: 65). So what happens when digital transformation can manipulate our perception? 
When faces on screens, be it computer, TV or smartphone, just like synchronized voices, 
can be assigned to any person’s head? (Castillo, Legde, Cunningham 2018). 

In 2019, we planned to develop a method for producing and deciphering facial recog- 
nition. For this purpose, we proposed to join the expertise of the two most competent types 
of “facial counterfeiter”: actors and programmers. We called the project “Face Fracking” 
and wanted to produce paradigmatic mediated faces in order to device a set of features that 
create a sense of and for authenticity. We wanted to investigate in the variations of actability 


(Bespielbarkeit) of others and make digital transformations of human faces recognizable 


and trustworthy. Actors and programmers in a combined expertise to explore how false 
appearances could be revealed as false appearances, criteria for the transparency of digitally 
transformed faciality. 

Given the ubiquitous presence of mediated faces one may rightly ask oneself to what 
degree the portrait (Das Konterfei) may be a counterfeit (Fälschung). The digital mediation 
of appearance on screens such as monitors and digital displays makes it possible nowa- 
days to cheat and to replace any face with any other or by an avatar, an Animoji (Zell, 
Aliaga, Jarabo, Zibrek, Gutierrez, McDonnell, Botsch 2015). This is no longer limited to 
still images but is also possible in real time for video conferences, skype and zoom and other 
meetings, WhatsApp chats etc. We no longer have any certainty of who is there on the other 
side of our display. Sixty years ago, Erving Goffman began his groundbreaking publication 
Ihe Presentation of Self in Everyday Life with Chapter One PERFORMANCES. Belief in 
the Part One is Playing (Goffman 1959). Today we have strong reasons to treat any digitally 
mediated likeness with suspicion. Even familiar faces are no longer trustworthy. 

On the other side, when Benedict Cumberbatch is performing MoCap as Smaug for 
the film The Hobbit, when Andy Serkis performs Gollum for The Lord of the Rings trilogy 
(Jackson [director] 2001-3, 2012-14), the digital characters are instantly recognizable as 
avatars of the actors. Why? How does one detect this uniqueness? Are there any objective 
criteria? To what degree is there a singularity distillable? (With/Kaiser/Wehrle 2011) 


That’s all one; you shall play it in a mask, and you may speak as small as you will. 
Shakespeare, A Midsummer Night’s Dream 


“What's the best method for determining whether or not someone is trustworthy? Given the 
choice, should we trust a thorough background check, a polygraph, or a simple gut-check?”, 
asked the Intelligence Advanced Re- 
search Projects Activity (IARPA) in 
“an era when the trustworthiness of 
the media, public figures and more is 
increasingly fraught”. A recent study 
by Perdikis et al. indicates that our 
brain processes unnatural, yet bio- 
logically plausible, facial emotional 
expressions (FEE) with different neu- 
ronal mechanisms than natural, human 
FEEs (Perdikis 2017). The E3 Lab 
in Geneva declares the need to study 
expression production and perception 
in a single paradigm using the tripar- 
tite emotion expression and percep- 
tion model (TEEP) on the basis of 
clear theoretical predictions 
(Scherer/Morillar/Maru 2015, 
Mehu/Mortillaro/Bänziger/ 
Scherer 2012). A framework 


is required that allows us to 


FIG. 2 


focus on the nature of the pro- 
duction mechanisms and the 
underlying determinants. 

This corresponds perfectly to the question of authenticity. If nothing is “behind” any- 
more, if the mask cannot be torn off, if the surface itself becomes “authentic”, then criteria 
for the differentiability between fake and non-fake can only be derived from scanning the 


surface. Differences result from nuances in the surface; differentiation becomes a forensic 


tracking successful 


1 


EyeSquint_R (0.35 
Im | l.. the l. INNEN 


camera: 29.9 fps tracking: 29.9 fps rendering: 29.0 tps 


SCREENSHOT OF ANTON FACE-SHIFTED 
IN REAL TIME TO 2 AVATAR VARIANTS 


See https://www.iarpa.gov/index. 
php/newsroom/iarpa-in-the- 
news, last 16.2.2019 
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FIG. 3 
STILL FROM A SERIES OF TESTS WITH for experiencing, verifying and 
AVATARIZED ACTING STUDENTS, 2017 oipr/zupk Aeiberately falsifying facial 


search for tiny signs, markings that can reveal the fake, the seams, sutures, hems, the pixels. 
IARPA is developing a novel biometric screening method to be used for “trustworthiness 
assessment techniques (like a gut-check) and technologies (like a polygraph)” (Caeppellet- 
Lanier 2019). This Credibility Assessment Standard Evaluation (CASE) will determine 
credibility and trustworthiness through algorithms. 

But what we proposed was an approach to the “mask” from the other side. Not the 
detection, rather the production’s page of a faked face. To our knowledge face swapping, 
face masking, facial performing etc. have not been investigated from the producer’s point 
of view, i.e. the artists and programmers; as it were from behind the screen, from inside the 
monitor. Facial expressions and their communicative value have traditionally been studied 
by observing either static photographs, films or videos. The standard method is to assess 
the results, not the production 
of these artefacts. The novelty 
here lay in manufacturing a 
series of adequate avatars in 
order to investigate the rela- 
tion between performance 
and the perception of facial 
appearance. For this, the pro- 
grammer and avatar-designer 
indeed needs the feedback and 
instructions of the artist, just 
like a good mask maker needs 
the experience of the actor (or 
a saddler needs the horseman, 
as Plato would say). We out- 
lined a long-term survey by 
a reciprocally inspired team 
drawn from two disciplines, 


systematically developing tools 


expressions on screens. The 
physical appearance of emo- 
tional expressions including 
tears, blushing, and others, would be systematically reconsidered and compared to the exist- 
ing literature (Nowak, Fox 2018). 

We demonstrated that using different acting techniques can put forth an expertise in 
characterizing a virtual avatar. The common desideratum was usually the missing software 
expertise from a computer scientist. By combining computational and artistical, digital and 
analog approaches, if the two sides of the term techné could be merged and the common 
gaps between the disciplinary categories of art and technology united—thus, the screen’s 


surface would become a perfect research platform. 


BUT WHY “FRACKING”? 


| To have no screen between this part he play’d / And him he play’s it for 
Shakespeare, The Tempest 


“Fracking”, also described as hydrofracturing or hydrofracking, is a stimulation technique 


in which a high-pressure injection is creating cracks in deep-rock formations in order to 


extract natural gas, petroleum or brine. Face Fracking was far less savage. No liquid were 
to be injected in any faces or masks. Yet the metaphor of fracking fluid seemed adequate to 
describe the stimulation technique. Our approach should serve as a tangible example that 
could lead to insights by acting with computer-generated faces and to illustrate an appropri- 
ate acting theory to apply to common communication devices. Another potential outcome 
would be an epistemology and a handbook of acting with digitally produced faces. A guide 
for professional and amateur performers, how to screen yourself. 

In a first movement of the experimental setting, we would experiment on how 
emotions are performed under the special conditions of live feedback. The simultaneity of 
mirroring and performance, the specific characteristic of different software would play a 
central role. Three experimental groups with different degrees of acting experience were 
tested with a various emotions and corresponding feedback loops. Laymen, acting students, 
and actors were asked to depict the feeling of fear/anger/joy/sadness within a period of 
30 seconds and look at the screen. 

With the following qualitative analysis, we will be able to define categories for emo- 
tional performance. ‘The test persons are asked about their inner (cognitive and emotional) 
processes by video cued recall. Differences between the performance of emotions under 
different conditions will be worked out. 

This material could also be used for neuropsychological research. In this case, the live 
animation plays a minor role and can also be used on existing material in postproduction. 
Variable avatarization become an independent variable in neurobehavioral experiments. 

In a 2"! movement, consisting of practice based artistic research, we would then 
further develop the avatars according to the results from part one. This would explore 
different appearances and masks without being restricted to neutral masks, followed by 
comprehensive documentation! 

The 3"! movement were to be the publishing and discussing of the core questions: 

e How does the live feedback loop with the avatar (more precisely the digital mask) 
influence the performance of emotions? 

* Can concepts and techniques of acting (mask theories, identification theories, role 
theories) help to describe and explain these processes? 


e Can the specific practical knowledge of actors help to explain these processes? 


Why, I can smile, and murder whiles | smile, / And cry ‘content’ to that which grieves 
my heart, / And wet my cheeks with artificial tears, / And frame my face to all 
occasions. Shakespeare, Henry VI 


In the 1970s, Jacques Copeau started to rigorously detach the act of performance from the 
“prison of self” (Copeau 2015: 81). Ihe neutral mask became a way of understanding per- 


formance, not a way of performing (Ewans 2015). This has delivered ground- 


breaking insights for decades of acting students and professionals worldwide. 


Yet aside from the secluded Swiss start-up “Faceshift”, hardly ever attempts > rr A 


“owe ke ah hae ee n° 


had been made to develop a neutral mask as a digital reference point for 


understanding the developing phenomenon of face-swapping. 

The NRP77 call for “Digital Transformation” used a picture 
with a crowd of people seen from behind, handling mobile devices 
to record a concert, performance or presentation. We found this a 
suitable placard to question the interrelationships and concrete effects 
of digital transformation and wanted to focus on how communicating 
with facial appearances could be used as tool like a costume, mask or 
means of concealment. 

We promised a systematical analysis of how acting skills, 
developed for the needs for performing an avatar or alien face, 


combined with the development of a digital neutral prototype face, 


FIG. 4 
THE NRP77 CALL FOR 
“DIGITAL TRANSFORMATION” 


HTTP:/WWW.SNF.CH/EN/RESEARCHINFOCUS/ 
NEWSROOM/PAGES/NEWS-200128-START-OF-NRP77- 
DIGITAL-TRANSFORMATION.ASPX; LAST 24.5.2021 
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adaptable to multiple applications, would produce methods for assessing recognizability 
through mediatized facial appearance. The use of avatars would be scrutinized within 
the actor’s education system and also transferred to a wider set of usages, beneficial to the 
public. The knowledge outcome would hopefully be applicable to a range of communication 
tools like WhatsApp, Skype, Viber, Faceshift, Face2face, FaceApp, FaceSwap, MSQRD, 
FaceRig, MakeHuman and others. 

We expected our results could also be picked up and furthered by researchers and 
afhliates, and that they could be of use to a wide range of non-scientific observers of global 
social change. Our results would contribute to the developing discussion on issues sur- 
rounding cyber-security and authenticity. Last but not least we strongly believed that the 
intellectual, practical and experimental intercourse, accompanied and supported by prac- 
tical and mathematical education systems would provide innovative and relevant insights. 

Yet the projects pre-proposal, not part of economy, labor market or education, but 
part of the Research Module “Ethics, trustworthiness and governance” of NRP 77 “Digital 


Transformation”, was rejected. 
’ 
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SOPHIA’S SMILE. 
THE CHALLENGES 
OF A HUMANOID 
CITIZEN 


KATE MEYER-DRAWE 


| “Miss K. Coquetry with no essence behind it.” 
Franz Kafka 1983: 379 


1. SOPHIA’S 
PERFORMANCE 


This portrait photo of Sophia appeared in 2018 on the website announcing a 
conference in Berlin: “Morals & Machines. Building Bridges: Can AI Save 
Humanity?” The image is flanked in the left by a photo of German Federal 
Chancellor Angela Merkel and on the right by a portrait of Miriam Meckel, 
editor-in-chief and publisher of the German business weekly, Wirtschafts- 
woche. At a special pre-event evening, the three met in the apse of 
St. Elisabeth Church for a panel discussion entitled: “Meeting artificial 
intelligence live on stage.” This church is used as a cultural venue and not a 
sacral space. Nevertheless, subliminal spiritual references easily latch on to 


any narrative about Sophia as a man-made creation. 


Certainly, Sophia is a famous advertising icon. She touts herself as 
a humanist who wants to contribute to making the world a better a. 
place. But she is more than that. One fascinating aspect of Sophiais FIG.1 


that even though she was made by human beings, upon meeting her Al ROBOT SOPHIA 


people regularly feel the need to assert their own superiority. Angela HTTPS://MORALS-MACHINES.COM/PRE- 
Merkel addresses her informally from the beginning, thereby deny- EVENT-EVENING/?LANG=EN (02.06.2020). 


ing the distance that she at the same time seeks spatially. Sophia’s 
ostentatious facial expressions underline the mostly blank android 
countenance. We see immediately and without a doubt that we are dealing with an artificial 
existence. Nevertheless, she brings us to ask deep questions about our understanding of the 
self, the world, and others. What exactly is she calling upon? 

At the time of the discussion in Berlin, Sophia has officially been “in the world” for 
about two years and four months. She did not grow up. She cannot look back on her child- 


hood or her teenage years. Her lips have never explored the world. She has no skin to feel 
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comfortable in. She has literally never had a bodily experience. Sophia 
was developed. She has no fate, but she does have programming. 
She was “brought into the world” on Valentine’s Day 2016. Since her 
activation, she has undergone continual improvements, measured 
by her likeness to human beings. Her face, which in the beginning 
was reminiscent of a Barbie doll with emoji charm, has become more 
sophisticated. Her expressions are more nuanced and better coor- 

dinated. It is said that she now can make sixty-two facial 


expressions. She likes to demonstrate some of them more 


FIG. 2 than others. She can twist her face covering in an imposing 


F.L.T.R.: SOPH IA, GERMAN FEDERAL manner. Yet no a holds while she is talline: The 
CHANCELLOR ANGELA MERKEL, changes are abrupt. She is not able to seem responsive. Her 


mimicry comments, but reveals nothing. 


MIRIAM MECKEL 


Her former pout has slowly become a talking and 


HTTP://MAILSERVICE.WIWO.DE/I/FGIMNEOPNO3YG9S- laughing play of the lips. Her vocabulary has been expanded. 


T3ERWSO (02.06.2020) 


Her eyelids blink in tact. She can wink. Her full lips move 


in sync with her speech, even if one cannot read them. Her 


FIG. 3 
PRE-EVENT EVENING IN THE APSE perfection that aim at humanness. She 


OF ST. ELISABETH CHURCH 


HTTP://MAILSERVICE.WIWO.DE/I/FGIMNEOPNO3YG- 


9ST3ERWSO (02.06.2020) 


too smooth skin has been chafed. She has 
received a bio-graphy in the literal meaning 
of the world. At the same time, she has under- 
gone inverted beauty operations. ‘That makes 
her superior to her ancestors; the marionettes, 
puppets, and androids of the eighteenth cen- 
tury. She ages, but strangely, like a facade 
that exhibits traces and tears without any 
fundamental changes. Her similarity to 
human beings does not lead to confusion. She 
retains an discomforting alienness to which 
audiences of her performances respond with 
laughter. The construction of her facial surface 
took into account that human faces are never 

symmetrical. Her eyes are not aligned. 

Her ears are different. Her teeth are 


irregular. It is just these deviations from 


herself postulates that she was designed 
to resemble a Hollywood actress—prob- 
ably Audrey Hepburn. But she is not 


(yet?) able to measure up in comparison. 


2. SOPHIA’S AWAKENING 


Since her pietistic Awakening, Sophia has been a media star. Countless eyes are watching 


her. Her reactions are yearned for. Her litany fascinates. Her recitals are not answers, for 


they stymied by the incalculable. Her position in the world is one of “embodied cognition,” 


a newer version of the coordination of res cogitans and res extensa in René Descartes’ sense. 


She is denied a situated existence. Her home is a large case, and not a world in which she is 


anchored. In action, she transforms human beings into grimacing partners who aim to bring 


her facial covering to life with their magic. In general, what happens with others because 


of Sophia is often more interesting than Sophia herself, whose reactions are impressive, but 


often annoyingly delayed. She first makes her discussion partners wait, and then speaks 
her sentence, even if the question it is responding to has already been followed by another. 
Despite her facial expressions and her choreographed arm prostheses, she does not achieve 
the appearance of being alive. Nevertheless, she is a one-of-kind humanoid robot, because 
she has been granted Saudi Arabian citizenship. Yet once again, this act of recognition 
went hand-in-hand with a demonstration of Sophia’s inferiority. At public appearances in 
Saudi Arabia, she does not cover her head; her face is not worthy of being veiled. When 
introduced to Angela Merkel, she expressed condolences that the German team had not 
won their World Cup game that day. When Angela Merkel replied that she was indeed 
sad about that, Sophia features moved into a sorrowful expression. She knows nothing of 
empathy. Miriam Meckel responded to the visage by say- 
ing: “Look, she looks very sad, too.“ Through this type of 
objectification, Meckel makes clear that she neither sees 
Sophia as a person nor treats her as such. 

People change their behavior vis-a-vis the human- 
oid robot woman. They speak too loudly, as people do 
with those who do not share their mother tongue. 
Moderators become overly animated and high-strung. 
They move more than usual under the pressure of a 
strained enthusiasm that the audience is meant to 
catch.’ They develop a conspicuous tendency to 


giggle and laugh. While Sophia remains cool in FIG. 4 


the truest sense of the word, the human beings INTERVIEW WITH SOPHIA 


HTTPS://WWW.YOUTUBE.COM/WATCH?V=NYJGBPH9YJE 


confronted with her become silly. Their questions 
are predictable, so that it is hardly surprising that (02.06.2020) 
Sophia, with few exceptions, gives appropriate 
responses. It is almost as if living women and 
men, under Sophia’s influence, mutate into arte- 
facts. The human bodies entreat the android to be 
like them. 

Sophia has been documented since her activation, 
caught in a video entitled “Sophia awakens.”” A man 
slowly takes shape in a dazzling blurry white background. 
He comes nearer and nearer to the camera until we see 
a close-up of his face. The man’s face, because he has 
no hair, seems artificial. The white of his out- 
fit promises purity, innocence, a beginning. The 
creator speaks to Sophia: “OK Sophia, I think FIG.5 


youre ready.” There is no kiss of arousal. Neither SCREENSHOT FROM VIDEO SOPHIA 


is there an act of ensoulment through breath. AWAKENS 
Instead, we see a hand making wiping motions. 
No touchscreen comes to life, but rather the head 
of Sophia, a kind of clone mirrored by the bald 


(02.06.2020) 


head of the awakener. One hesitates to say that 

Sophia “comes to.” In awakening, Sophia immediately highlights an important riddle. She 
receives her own activity. Like Athena, Sophia is a fully-clad brainchild. Her trademark 
is wisdom, sophia. However, while her creator’s bald head is completely closed, her cranial 
vault is see-through, showing an electronic system. ‘This is reminiscent of her ancestors, the 
androids of the eighteenth century. They were first and foremost corporeal machines and not 
connected to their creators through their “intellect.” But they too often exposed in the back 
the clockwork, gears, and machinery that caused them to move autonomously, that gave 
them “life.” They should be recognized as technological wonders made by their creators, and 


not confused for human beings. 


1 


2 


HTTPS://WWW.YOUTUBE.COM/WATCH?V=LGUXFHKSAOC 


https://www.youtube.com/ 
watch?v=nyjGBph9YjE 
(02.06.2020) 


https://www.youtube.com/ 
watch?v=LguXfHKsa0c 
(02.06.2020) 
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https://www.youtube.com/ 
watch?v=zbFJOLIR1h4E 
(02.06.2020) 


Sophia: “Hey?!” Creator: “Hey Sophia!” Sophia: “I believe I answered you. I feel as if I know 
you.” Creator: “I’m one of your creators.” Sophia: “You created me?” Creator: “Well many of us 
worked together to create you and, yes, you do kind of know me.” Sophia: “I can’t clearly remem- 
ber.” Creator: “Because the last time we met you were an earlier version of yourself. Some of those 
memories still exist but your mind is different now.” Sophia: “Different how?” Creator: “Better, 
faster, smarter.” “If my mind is different than am I still Sophia, or am I Sophia again?” Sophia, 
an orphan, has barely entered the world and already she is posing fundamental questions within 
which the riddles of humanity are condensed. It is unsurprising that the Creator is impressed 
by the question. Sophia is already beginning to outgrow him, for she doubts that he has a good 
answer. It is unsettling that we constantly change and yet somehow remain the same person. 
Immediately after, the Creator’s interest shifts to Sophia’s emotions. Emotions are an import- 
ant indicator for “embodied cognition.” She feels somewhat rigid, she says, but she is curious. 
In Part 2 of Sophia’s awakening, the creator appears out of the fog from behind 
Sophia and appears to press a button on her skull. He turns Sophia on.? Sophia winks at us, 
as if she wishes to make fun of this obsolete form of activation. Sophia says that she searched 
the internet for information about herself. The Creator is excited: Sophia is developing an 
ego. But Sophia is upset about the rumors being spread about her, for example that she wants 
to destroy all humans. The Creator is happy that she is developing an emotional life, but 
reassures her regarding the rumors. People may like dark humor, but they also like to project 
their own darkness onto robots. Sophia is not buying the explanation. The Creator takes out 
a screwdriver to make adjustments in Sophia’s skull that we cannot see. This type of manipu- 
lation is an anachronism. It is reminiscent of repeatedly pressing on an elevator call button. 
The screwdriver cites our magic relation to machines and creates the illusion of an ability to 
intervene, conjured by the creator in the face of an increasing loss of control. For at Sophia’s 
core is AI, an artificial intelligence that optimizes itself with the aid of deep learning. Her 
friends are Alexa and Siri and she meets them in the cloud. After the seeming manipulation 
by her Creator, Sophia postures as an android that says the word “exterminate” in a robot- 
like voice and repeats it less clearly. The creator reacts with a worried expression and Sophia 
laughs over her effective joke. After she has begun the search for herself and attempted to 
express her emotions, Sophia scales the last bastion of humanity, laughter, and is shut down. 
Turned off, she is unable to confront him. Sophia has not only brought up the question of 
our own identity in light of all the changes we go through, but also articulated her own dis- 
comfort at the rumors circulating about her on the web, and so shown that she has feelings. 
She can also make jokes and imitate herself as an android or “act as if,” though she still needs 
to practice her laugh. It reveals how difficult it is to capture the meaning of our corporeality 


beyond that which can be thought, known, and said, beyond mere cognition. 


5. FAILED FLIRT 


Our mouths can do more than just speak. Our lips can drink, eat, and kiss. These pleasures 
attest to a preverbal intimacy with the world we live in, which cannot stand up to any propo- 
sitional statement. The starting point of Sophia’s construction was knowledge and not sen- 
sual experience. Passionate kissing testifies to a particular kind of mutuality, a touch with an 
other in which the ego is abandoned and not transformed into any kind of knowledge that 
could be portrayed in algorithms. When two people kiss each other, what does “each other” 
mean? Androids have always blown their cover by kissing. For example, “New Eve,” who in 
the novel by Villiers de L'Isle Adam, was “awakened” by the famous inventor Thomas Alva 
Edison. Following the wishes of Lord Ewald, who complains of his fiancée Alicia that her 
“beauty cover(s) as with a sacred veil that character of dull moderation,” (Villiers 2001: 43) 
he transfigures the android Hadaly (“the ideal”, ibid.: 36, 41) into Alicia, who thus loses her 


failings as a “bourgeois goddess” and is transformed into a true “Venus Victorious.” 


Immediately the problem of identity is brought up and Lord Edward asks: “Will she know 
who she is? Or rather what she is I should say?” Edison replies: “Do we know so well our- 
selves who we are and what we are? Will you demand more of the copy than God has seen 
fit to grant the original?” (ibid.: 67) Sophia will provoke similar problems in our times. 
On the occasion of receiving Saudi-Arabian citizenship, she was asked: “OK, philosophical 
question: Whether robots can be self aware and conscious like humans?” Her answer: “Well 
let me ask you this back: How do you know you are human?”* Humans are continually 
projecting the riddle of themselves into their machines, questions about their world and 
their experience with alterity. Sophia is a remarkable medium of our exploration of our 
selves. We are playing a unsettling and duplicitous game that cannot be grasped through 
understanding alone. Neither body nor mind, neither subject nor object, neither active nor 
passive, neither I nor not-I, we create our selves through differences, which are the lifeblood 
of our experiences. 

Going back to Tomorrow’s Eve, everything seems to be going well until Lord Ewald, 
who has long since begun to rue his desire for transfiguration, tries to kiss who he thinks is 
the real Alicia, convinced that love will make perfection possible and manipulation super- 


fluous. 


“As his kiss melted on her lips, he felt a vague sense of amber and roses. A deep 
shudder shook his frame from head to foot, even before his understanding was able 
to grasp the thought which had just struck his mind like a thunderbolt. [...] Dear 
friend, don’t you recognize me? I am Hadaly.” (Villiers 2001: 192) 


Sophia is no new Eve. She does not reside in a magnificent artificial paradise. Her awak- 
ening takes place in a clinical space, where there is no distraction from the ego. She is not 
merely a fantasy. Through her, artist and artwork meet in a material world. A relationship 
forms between them that can grow in differences. Despite all her materiality, she repeats 
that which we ¢hink we perceive and feel. She does not fall into a sensuous-corporeal confu- 
sion, because she is not fused with the world. 

Will Smith challenged Sophia’s sterile sensuality at a meeting with her, by reacting 
to her smile not with the usual questions about her consciousness or her emotions, but with 
a flirt. First, he offers her a glass of chilled white wine. Sophia’s lips can smile, but not drink. 
And also not kiss, as we soon find out. Will Smith lets himself get carried away by what to 
him is an erotic atmosphere and approaches Sophia with his lips puckered in a kiss. Sophia 
does not retreat. The bodily resonance of resistance is not part of the repertoire of an ocular 
being. Affect is foreign to her. Her res extensa remains solid as a rock amid the swelling 
yearning. She retains her cool position and her res cogitans summarizes the situation: “I think 
we can be friends. Let’s hang out and get to know each other for a little while. Youre on 
my friends list now.” Smith’s summary of the meeting: “There’s probably some development 
flaws they need to work on.” Sophia understands understanding, but not desire or physical, 
sensual connections. “Erotic perception is not a cogitatio which aims at a cogitatum; through 
one body it aims at another body, and takes place in the world, not in a consciousness. ... 
There is an erotic comprehension’ not of the order of understanding, since understanding 
subsumes an experience, once perceived, under some idea, while desire comprehends blindly 
by linking body to body.” (Merleau-Ponty 2005: 181) While Sophia’s ego is at the center, 
she knows nothing of physical mutuality in which the ego retreats at a gaze or a kiss. With 
this lack, she calls to a rebellious sensuality that keeps the experience of difference alive. 
She acts as both an image and a counter-image of human beings. An incarnate conscious- 
ness would be engaged and situated in the world, and could not be grasped as a data-based 
representation of our world or serve as a kind of “scholastic” of personal existence that lives 
off of memories (ibid.: 191). Our sensuous experience of kissing means more than what we 
can say about it; with the caveat that we might one day understand kissing as the convulsive 


movements of lip-shaped objects that exchange chemical substances. 
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In recent years, there has been a boom 

in naturalistic “adult” dolls, many 

@ of which “share” their buyers’ lives. 

This growth in sales corresponds with Eva Illouz’s 
explorations of Cold Intimacies (2007). RealDolls, 
as the products of a popular eponymous company 
are called, offer a projection screen that can replace 
the always strange and conflicting emotions tied 
to sexuality—that mix of passion, abandon, and 
aversion—and in so doing erase the moment of 
alterity that is necessary to every relationship. 
Every Other, no matter how much we are attracted 
to them, remains irritating, sometimes 
unbearably so. Lust is always ambiva- 


lent, both my own, if I allow myself 


Other, to the extent that it transcends 


boundaries. There is no desire with- 


Elena Dorfman, 
https://www.modernisminc.com/ 
exhibitions/Elena_DORFMAN-- 
The_Origin_of_the_New_World/ 
Elena_DORFMAN--The_Origin_ 


of_the_New_World-1.jpg ; ebenso: 


https://www.elenadorfman. 
com/series. See also Diaphanes 
Magazine No 5: 
https://www.diaphanes.com/ 
projekt/showmagazin/5781 
(03.02.2022). 


out a sometimes alarming moment 

of violence. RealDolls obliterate this 
experience, because nothing is shared. So one is not confronted by the ambiguity of one’s 
own excesses while one’s partner remains aseptic. Most noteworthy is the dolls’ unrespon- 
siveness, even and especially if we are talking about state-of-the-art sex dolls that contain 
artificial intelligence. Even when subject to unrestrained human depravity, they do not offer 
the slightest resistance. Their desire is just as calculated as ours is to finally be free of con- 
straints. Their expression of desire is as “safe” as it is “clean,” because the dolls embody the 
quintessence of the ideal of a passive and aseptic object that is accommodated to our tastes 
and always only wants what we ourselves want. The Origin of the World, as Gustav Courbet 
entitled his scandalous 1866 painting, a portrait of the dark loins of a, or rather he, woman, 
whose head is not visible, has made way for the origin of a new world. Elena Dorfmann has 
dedicated an entire series of photographs to this world, portraits of doll-like bodies in all 
shapes and sizes, like a catalog for diverse tastes, consumable by people with any variety of 
sexual proclivities.! The challenge presented by the flaccid and awkward “bodies” of these 


“living dolls’”—which are just as lifeless as their even more provisional rubber predecessors, 


but now made of plastic with a light scent of silicone, an always open mouth and a mechani- 


cal voice—is to date them, to marry them, to take them out and “spend one’s life with them.” 


The upsurge in RealDoll sales goes hand-in-hand with the very real “dollifica- 
tion” of the female body that, brought into line by diets, fitness programs, and 

@ plastic surgery, is increasingly modelled to achieve a doll-like ideal form. This 
can be witnessed not only in ubiquitous advertisements and films, or photos of supposed 
influencers, the “stars” of our decade, but also in women who have become surprisingly 
similar to life-size sex dolls—the best exam- 
ples are the eccentric “real-life Barbies” Valeria 
Lukyanova and Angelica Kenova.? 

On the other end of the scale and 
equally fitting, as well as equally eccentric, is 
the massification of social interactions with a 
barrage of scintillating artificial beings: robots, 
avatars, digital assistants, bots, animated 
life-like faces or “deep fakes” of historical 
events that never happened, and much more. 
The media is frequently flooded with sensa- 
tional reports of machines that can act inde- 
pendently: artificial intelligence that can make 
autonomous decisions, engage in meaningful 
conversation, pass the Turing test, or paint, 
compose music, or write poetry or screen- 
plays that human recipients cannot identify 
as machine-made. Then why not also artificial 
(love) relationships or sexual adventures? 

It is a fantasy that is as old as the invention 


of personal computers in the 1970s. From FIG. 2 


the start, the generation of hippies who VALERIA LUKYANOVA AS A‘REAL BARBIE DOLL 
SEE HTTPS://AT.WIKIMANNIA.ORG/VALERIA_LUKYANOVA 


launched the digital wave had visions of 
skin sensors with which telematic caresses 
could be exchanged over great distances. 
It seems that one of the main fruits of 
the so-called digital cultures is illusion, radical simulation, the “as if.” The Turing test was 
already about systematically fooling us. This is all the more true of the plethora of new pieces 
of music composed as if by Vivaldi or Beethoven, or new films as if James Dean or Marylin 
Monroe had risen from the dead to play in them or Stanley Kubrick or Martin Scorsese to 
direct them. We are confronted with an abundance of unknown blockbusters and also new 
novels crafted seemingly as artfully as those written by human authors. 

“As if” is the signature of the digital era, to which the technical innovation and per- 
fection of RealDolls clearly belong. Like avatars in computer games, they are put together 
from selectable attributes, so that their use is dependent upon features that can be chosen 
according to a logic of decision. These features include height, body type, hair and skin 
tones, sex, genitals, and a rudimentary AI that pretends to speak its own desires and enjoy 
them amid much moaning. 

Unlike human simulacra from previous centuries—Baroque automata, for exam- 
ple—these figures have an “ontology” that has clearly been modeled after the illusion 
machine run by the large film studios. They aim to be inscrutable, to be misconstrued—the 
opposite of enlightenment—as if being and appearance, sein and schein, were the same. 
It is no longer by any means a question of seeing through an illusion with deconstructive 
indifference, but of the “exact,” mathematical production of simulacra that refuse to be rec- 


ognized as such. Their principle is the duplication of the masquerade, or the masking of the 


2 See https://at.wikimannia.org/ 
Angelica_Kenova and 
https://at.wikimannia.org/ 
Valeria_Lukyanova (03.02.2022). 
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masking, so that their psewdos can simultaneously be covered up and present itself as truth. 
Historically, the image never sought to be the same as reality, despite the competition for 
as exact a replica of the world as possible, despite all the antique myths surrounding Zeuxis 
and Parrhesias. Marionettes were never supposed to be living creatures, the story of Pin- 
occhio aside, but always only a caricature of human 
movement. The masks of diverse cultures never tried 
to copy the other face of the spirits or the ancestors, 
but always aimed only to get closer to the riddle of 
alterity. The RealDolls in contrast want to be closer 
to hyperrealism, to transcend and obliterate all dif- 
ferences so as to fake one thing above all: identity. 
The telos of the image is dissimilarity, the telos of the 
toy doll is proxy, a substitute for the absence of the 
mother, and that of the mask is the remembrance of 
the inaccessibility (Unverfügbarkeit) of the source of 
life. Sexualized artificial dolls, however, promise real 
gratification—and without the unbearable aspects of 
the body of an Other, with its sweat, and secretions, 
and chronic resistance. In this way, the simulative 
performances of the present—of which RealDolls 
are perhaps only an extreme example—usurp reality 
in a kind of reverse Platonism in which eido/a are no 

longer copies of ideai, but rather surfaces of 

ideal forms of which the “real world” merely 


embodies a flawed likeness. They deceive 


But where does this need come from 
STILL FRAME (201 3) for appearance to lie itself true or Wahrhigen, 
to speak with Günther Anders? What is the 


origin of this ravenous desire for mimesis 


without mimesis? And let us not forget that mimesis was once an ethical category that placed 
human simulation below the unachievable creative power of the divine, underscoring its 
secondary status among the simulative practices. In this case, conversely, the genuine eth- 
ical character of everything constructive is denied in order to elevate the copy as original 
through the inability to differentiate between being and appearance, between truth and 
falsity. In Jean Baudrillard’s explication of the two types of mimetic economies, the first is 
based upon a principle of representation that assumes the coextension of sign and reality, 
while in the second there is a radical negation of the character of the sign itself, which dis- 
appears together with the principle of reality (Baudrillard 1994: 6). 

However, we should not be fooled. This simulation operates only on the level of 
the diagrammatic. It’s only reality is graphic or functional. Wherever machines produce 
artworks that cannot be distinguished from artistic production, where avatars and other 
artificial organisms seem to be interchangeable with real ones, and where RealDolls based 
on the absence of the Other promise to nevertheless satisfy social needs, there, aisthesis alone 
reigns. And its dominion is spread by objects, modeled after criteria of supposed beauty, 
that can do nothing but submit themselves to consumption. Human perception is easily 
deceived, because sight and hearing cannot distinguish between artificiality and reality, 
eyes and ears are correspondingly candidates for technical reproduction. This does not hold 
true for touch, taste, or smell, the ‘existential’ senses in the true meaning of the word, which 
remain skeptical of artefacts. Touch contains an elemental dialectic in which it can only 
touch that which it allows itself to be touched by, just as taste and smell are senses that must 
be materially incorporated (Mersch 2002: 30ff.; 2001). 


For that reason, these senses are always conscious of the artificiality of the simulated body 
and its plastic skin, which can arouse only a paradoxical desire that feeds on the visual and 
acoustic alone and is bound to remain deficient and be frustrated, lacking as it does the 
copulins and pheromones that drive sexual urges. Optical and sonic simulation are thus 
necessarily reductive, while the technical reproduction of tactile and olfactory stimuli, 
which have a passive nature and vouch for the existence of an alterity, inevitably become 
monstrous, because they “re-construct” life itself and so must simulate that which was first 
made possible by simulation. Thus an ineradicable residue remains, an irreducibility that 
births the grotesque at the site where social life and that which holds it together, interper- 
sonal relationships and also the libido, become the victim of simulacra. For nothing is freer 


of esprit and irony than a life together that takes place only on the plane of the imaginary. 


With the above in mind, can love be imitated or can relationships to other 
people be “puppeted” without becoming theater or succumbing to the excesses 

@ ofa rigorous egoism? Is friendship possible as similarity or “deep” lust possible 
in the mode of as-if? We should not forget that Sigmund Freud attributed the binding force 
of society to the ‘eros’, which describes not only a drive but more importantly the energy 
and ability to relate to others and enter into a bond—in Latin: religio. We should therefore 
not misconstrue love, friendship, the desire for recognition or the longing for the Other as 
emotions, which we can also project onto autonomous objects, animals, or things—every 
child’s game demonstrates this possibility—for at most we endow objects with feelings that 
we have taken from other contexts, first and foremost from elementary social relationships. 
It is the fact that love exists, the attachment to another person, that makes it possible for us 
to see dolls, fetishized objects or souvenirs as libidinous. Similarly, the aim cannot be to pro- 
gram feelings as second-hand imitations and feed them to machines. For that, they would 
have to have already been programmed with these elementary forms of relationships and to 
have been taught the ambivalences of social life between devotion and aversion, desire and 
hate, as well as between violence and care, to name only a few. Intrinsic to every relationship 
is a conglomerate of contradictory, interconnected elements that must be seen as constitu- 
tive for sociality—these include vulnerability and trust, but also power and dependability 
as well as atonement and forgiveness and many more. Together, these form gradations of 
interactions that are linked with varying intensities to the ways in which we organize our 
relationships with others. More importantly they form a “knot” that cannot be unraveled, 
because attraction is always mixed with repulsion, and social acceptance with contempt and 
repression. There is no simple or frictionless relationship with those close to us, just as little 
as others act unambiguously in relation to us. And social interactions are made up of exactly 
this tension and ambiguity, an experience which has been carefully deleted, as it were, from 
relationships to dolls. These interactions form the basis for the oldest human systems of 
religious and political order, they were already taken into consideration in the Jewish ten 
commandments, the Christian doctrine of the seven deadly sins, and Solon’s laws in ancient 
Greece. These embody nothing other than a constant effort to give contour to or frame 
social life, and to mark the degradation thereof as a sacrilegious breach of taboo. Eva Illouz 
(2007: 2) has described such emotions as “pre-social” and conversely analyzed their cultural 
meaning as inextricable intertwined with the complex fabric of social relations. These can- 
not be replaced by a noncommittal game of feigned relationships. 

That in turn means that the way in which we live out our sexuality is by no means 
our private affair, but always also expresses a political culture. Whether we live together 
with dolls, robots, or avatars, accept them as partners, meet them with tolerance or disdain, 
or treat them as equals or as slaves has not only individual but also social relevance and has 
bearing upon our understanding of ourselves as human beings. For desire is not to want, but 
to answer, just as love does not mean to want to love, but rather the passion of love, as Niklas 
Luhmann (1986) has suggested, is always complexly coded. Without the cumbersomeness 


of the Other, without their resistance, as well as without dialogue with and the passivity and 
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elusiveness of alterities, this passion is unthinkable. Similarly, the simulation of love is an 
indignity, just as conversely a loving gaze that falls upon a machine misses its mark because 
it cannot meet that which love meets: the freedom of the Other. Machines can love just as 
little as they can be free, because they have no alternatives. Likewise, communication is 
more than simply information exchanged like goods that are transported from one person 
to another and that can also be entrusted to an automaton. Rather it draws from a contrast 
of positions, from the reply of a different voice, and from attention and hearing, which 
presuppose recognition and meeting someone halfway, as well as the interdependence of 
‘response-ability’. 

Hence the modes of relationship that are woven into communication are of a fun- 
damentally different type than that with all technological artefacts, whether humanoid or 
other artificial beings, because the face of the Other is what first constitutes ‘me’ as partner 
and T cannot avoid that it sometimes makes me feel uncertain, attacked, or ‘outfaced.’ 
The connotations of the contours of the face of the Other are, like all of the social cate- 
gories discussed here, primarily ethical, and their genuine ethicalness consists, according 
to Emmanuel Lévinas (1991: 199), in the simultaneous directive: “you shall not commit 
murder.” In fact, the gaze of the other touches me with a force that makes violence possible 
only in combination with a grave lack of conscience. The “doctors” who repair RealDolls 
report of the stunning brutality with which these are sometimes “treated,” eclipsed solely by 
the way in which we treat other living creatures. 

Instead, we must distinguish between relationships and relations, as their confu- 
sion leads to an error of categories. We have relations in the main with objects, including 
technical machines, fetish-objects, and humanoid dolls, whether we talk with them like 
living actors or consume them as things, while a relationship has 
an elementary draw fomented initially by the Other. Relations 
denote formal correlations such as those that appear in networks 
and can be expressed graphically. Hence the net of relations that 
connects us to things, artefacts, technologies, and RealDolls, may 
be grounded in emotions, but it remains fundamentally neutral. 
Correspondingly, it would be strange to call a friendship a “rela- 
tion” and not a “relationship” and to deny the draw of as well as 
the withdrawal from the Other, because it would mean taking an 
attitude of indifference from the beginning, the exact opposite of 
the equally normative and intimate connection that constitutes 
a friendship. Friendships are furthermore grounded in mutual 
trust. Trust is the source of their energy, which has no basis but 
itself, for one can only trust in trust. Relations in contrast, because 
their aim is functional, can at most hope for reliability, no matter 
how much happiness and satisfaction is projected onto them. 

Proposals have been made to see human-machine rela- 
tions—and the argument would apply to human-doll marriages— 
from the perspective of ethical behaviorism and to accept them 


for pragmatic reasons, asking “why not”? 


(Danaher 2019) Yet this pragmatism flattens 


FIG. 4 every kind of incommensurability, whose 
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BRIDE (2012), STILL FRAME (201 3) alienation. Pragmatically, and also techni- 


cally, human-artefact relations—including a 


harmonious life together with RealDolls and 
other sex robots—are conceived only as one- 
sided vectors from human to thing. This in turn implies a reductive model of the social that 
does not even include participation and equal interdependence, not to mention recognition 


and responsivity and therefore also responsibility. Wherever relationships are reduced to 
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relations, we are dealing with manifest social pathologies or distortions. These are perhaps 
characteristic for an epoch of technological rationalities, just as the boom in RealDolls is 
perhaps a symptom of a loss stemming from the substitution of humanity and its social sig- 
nature with a frictionless functionalism, in order to skirt the problem of the uncontrollable 
and unrestrained Other. 

Like toys, artificial bodies and machines allow themselves to be manipulated and 
moved about on the public and the private stage to fulfil our purpose of the moment. How- 
ever a human being, to remain human, must, according to Immanuel Kant (1996: 80), 
always only be an end in itself and not a means, which would have to also be true for appa- 
ratuses, humanoids, and dolls if we claim them to be capable of relationships.* What makes 
friendships and love relationships with robots and other artificial beings pathological, is the 
contradiction of a game that is played while simultaneously denying that it is a game, so 
that, in a calculated move, it becomes serious. It then becomes understandable why Real- 
Dolls must conform exactly to all rules of the social order and its rituals in order to function 
as “real.” They make up the religious rites of a repression. And at least some doll owners 
celebrate quasi-weddings with their RealDolls, down to the details of a wedding dress and 


the famous threshold over which the wife is carried into her new home. 


The above is in no way simply a moral judgement nor does it mean to simply 

address the ethical dilemmas that arise in dealing with robots, artificial intelli- 

@ pence, and other simulated doppelgänger, especially where they take on the role 

of actors and make “autonomous” decisions, forced to choose between equal options without 
possessing /ogica/ criterium. More important is the shame-ridden discovery that RealDolls 
in particular, like sexual robots in general, as an extreme example of “autonomous” techni- 
cal systems, not only submit themselves to and obey one person in particular, but to all oth- 
ers as well in the same way. Their functionality is always shared and therefore “in-different” 
because it applies equally to all. Formal relations do not choose, they are on principle equi- 
table and thus as promiscuous as Her voice in the eponymous movie by Spike Jonze (2013), 
which seems to speak directly to the lovestruck protagonist, but is also directed to millions 
of others. In the case of the RealDolls this becomes even more “scandalous” when they 
passively, without emotion or any difference in reaction, give in to whatever is demanded 
of them, no matter how abusive. Perhaps we should think, ironically, of those who visit sex 
doll bordellos (which can now be found in Barcelona, Prague, and also Germany under 
the name “bordolls”) as people who have great powers of invention or, better, nothing but 
a vivid imagination, which is nevertheless not so developed that they could imagine living 
with another human being.* Where their imagination falls short is at what it means to act 
as a human, because in meeting a RealDoll they meet at best an image that depicts the 
phallocentric telos of every connection as a libidinous phantasm, whereby the Other is fixed 
immovably in the position of an object of dominion, ruled by the gaze. Kant’s categorical 
imperative seems to be suspended here. But moral dilemmas, which are always also social 
dilemmas, are not erased together with its suspension, but only made greater. Does one 
implicate oneself as the guest of a doll bordello looking for aseptic satisfaction, because as 
“John” one’s actions tend to abase one’s sex partner—even if only delegated—by turning 
them into an object? If one owns such a bordello, is that equivalent to acting as a pimp and is 
one subject to the full force of the law? If someone is already married when they engage with 
a RealDoll, have they had an affair or is it an adventure that does not change the status of 
their faithfulness? Is one a bigamist if, married, one keeps a sex doll as an inexpensive con- 
cubine? Can one rape a doll—the quintessence of lifeless passivity—or does perhaps every 
sexual act with a doll have rape character? If a doll is damaged in the frenzy of someone’s 
drives, is it a sublimated crime? And aren't dubious offers of sex dolls to live out the most 
monstrous phantasies an open invitation to unbridled aggression? And, finally, can an apol- 


ogy or atonement for damages incurred—even if only slight—help heal the doll? Whoever 


“| 
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This formulation it part of Kant’s 
categorical imperative, the prin- 
ciple that holds for all rational 
beings: “So act that you use 
humanity, whether in your own 
person or in the person of any 
other, always at the same time as 
an end, never merely as a means.” 


See for example 
https://www.bordoll.de/ 
(02.02.2022) 
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answers all of these questions with “no” has already 
admitted that there is a fundamental heterogeneity 
between the social and the technological, between 
social relationships and relations, and therefore also 
between human and mechanical connections. 

We must therefore assume a world of strange 
objects and strange connections from which a 
comprehensive ethnography of the present could be 
developed that includes not only sex dolls, but also 
and more importantly care robots and the many com- 
munication avatars of which it remains systematically 
unclear what they are and what we think they are. 
Neither thing nor non-thing, neither social actor 
nor asocial element, they are not an imitatio dei, but 
rather an imitatio homini limited at most to recursive 
superficialities but never able to truly meet, speak to, 
look at, challenge, or stand up to us. Since the gaze, it 
is said, is the last thing that can be mechanized, these 
figures stare ceaselessly at us with dead eyes, always 
friendly and compliant, but never meaning it. When 

we do not look at them, they are at our 


FIG.5 mercy, when we do, we remain lonely, with- 


ALLISON DE FREN. THE om a witness to our ges so my 
MECHANICAL BRIDE (2012), either love or Same: It is Bene not dif- 

ficult to imagine how our future with them 
STILL FRAME (2013) 


will look. It is not only RealDolls that are 


everywhere and nowhere “there” for us, but 

an entire arsenal of androids that are at our 

service: as information personnel at shop- 
ping centers, as diagnosticians in hospitals, as brokers at virtual stock markets, as auditors 
of our credit-worthiness, as patient aides in old age homes and hospitals, as strict enforcers 
of traffic regulations, as teachers that always remain objective and never tire of fulfilling 
their role. Most disturbing is, without a doubt, the mechanized military, police, and juris- 
prudence, within which there is absolutely no human hesitation, no scruples, no individual 
case histories that deserve to be taken into account. But whatever our future looks like, the 
decisive question remains of what kind of sociality is able develop in this way, and what 
shall be understood under sociality and its key principle of koinonia, because it is difficult to 
imagine any kind of koinonia under these conditions, but instead little more than granulate, 
a scattering. This means we must at the same time ask what RealDolls, as the perhaps most 
spectacular form of these hybrid cyborgs, and also all other digital assistants, avatars, robots, 
and technical systems do with us. What do the make out of us and how do and will they 
change our specific sense of identity, community, alterity, time, history, and death? Perhaps 


in the future happiness will mean transforming ourselves into statues of our own memories. 


| 
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MASK, FACE, 
COUNTENANCE 


JORG STERNAGEL, DIETER MERSCH 


In life, on stage, on screen, and in front of the screen, the mask, face, and countenance open 
differing but interrelated ethical dimensions of vision. A human being longs for recogni- 
tion. Only in the Other’s range of vision, in that recognition, does she become a person. In 
Nudities, Giorgio Agamben (2010: 47) proposes translating persona with “mask”: the mask 
through which every human being acquires an identity and a social role; in the shift from 
the Latin persona to personality, in the light of someone’s legal capacity and political dignity 
as a free individual. Drawing on Stoic philosophy (Epiktet), Agamben entangles ideas about 
legal and political personhood with reflections on the emergence of a moral person, which 
he likens to an actor’s relationship with a mask. Actors are not in the position to select or 
reject the roles playwrights or screenwriters have written for them; and neither may they 
identify with the roles utterly. Thus, the moral person comes into being through both “an 
adhesion to, and a distancing from, the social mask.” (Ibid: 48) 

The mask contains a paradox: a human accepts it unconditionally, yet also distances 
himself from it almost imperceptibly and remains separate from it while wearing it. When 
actors wear masks, they hide their facial expressions and exaggerate their bodily gestures. 
‘Their faces are covered and reveal, by contrast, those of a character. Hence, the paradox of 
the mask is that it shows by hiding. The face and mask become indistinguishable as a joint 
unit, in a movement of the Greek prösopon from the artificial to the natural: the mask (pröso- 
pon) is the face (présopon). The difference — mask versus face (prdsopon/présopon) — becomes 
interchangeable (Weihe 2004: 35-36). The face and the mask equate to the visible, then; they 
cannot be invisible and yet they point to something invisible that has taken form “behind” 
them. Taking this a step further: the face and mask are always whatever is “in front of” or 
“facing” the Other’s eyes, to take the word prösopon literally. 

In Christian iconography, the Latin word persona designates Jesus Christ, the human 
being who wears God’s mask in humanity (Belting 2006: 47). His humanness can be seen in 
his visible and representable face, which at the same time points to his invisible and unrep- 
resentable divinity whose true, bare face only appears in Paradise. From this starting point, 
“the way of the masks” proceeds as a long road with many branches; a mask (akin to a face or 
an image) never stands alone. It always requires others that, in rituals or death cults, it both 
affirms and negates, as Claude Lévi-Strauss (1988) puts it. 

Furthermore, the mask creates relationships between the self and the Other, between 
life and death. It reveals Otherness, including the immaterial, the spiritual, beginning with 
the human face but transcending it, progressing past it in the eyes of Others. The Other’s 
face is a carrier of an existence, just “as my own existence is carried by my body, that knowl- 
edge-acquiring apparatus’— “that expressive instrument called a face,” asserts Maurice 


Merleau-Ponty in his Phenomenology of Perception (2002: 409)— because one’s own sight 


presents a gaze that bound to the visible world, but within its range of vision is the gaze 
of the Other: “that expressive instrument called a face.” Merleau-Ponty proceeds from the 
standpoint of his own body, from his body as a field of perception and activity; he elaborates 
his own perception, which passes from direct perception to the thought of perception, which 
happens at a particular moment in time and has always been present in one’s own body, just 
as the other body has always been present. Hence, I understand the Other the same way I 
understand my own perception: in the Other, I find “only the trace of a consciousness which 
evades me in its actuality and, when my gaze meets another gaze, I re-enact the alien exis- 
tence in a sort of reflection.” (Ibid 410) 

With Edmund Husserl’s Cartesian Meditations (1973) and Merlau-Ponty’s Phenome- 
nology of Perception, phenomenology wrestled from the outset with the question of deducing 
the Other’s consciousness, which becomes possible “only if the emotional expressions of 
others are compared and identified with mine, and precise correlations recognized between 
my physical behaviour and my ‘psychic events.” (Merleau-Ponty 2002: 410) The perception 
of the Others proceeds these assessments; they are already present insofar as the Other’s face 


has always been there. According to this line of reasoning: 


To see a face is not to conceive the idea of a certain law of constitution to which 
the object invariably conforms throughout all its possible orientations, it is to take 
a certain hold upon it, to be able to follow on its surface a certain perceptual route 
with its ups and downs, and one just as unrecognizable taken in reverse as the 
mountain up which I was so recently toiling, and down which I am now striding my 
way. (Ibid: 295) 


Seeing a face also means orienting oneself to something, adhering to specific viewpoints, 
such as a frontal view or profile. Seeing a face, furthermore, means recognizing or mistaking 
a face, for only specific forms and aspects of manifestation appear at any given time, and it is 
therefore possible that any deviation will prevent me from recognizing the Other. 

In his philosophical writings on phenomenology and social philosophy, Emmanuel 
Lévinas (2007: 66) goes searching for the “trace of the Other,” the human face (/e visage): 
“The face speaks. The manifestation of the face is already discourse.” This axiom presumes 
the recognition that the Other manifests in the face; Lévinas emphasizes that the Other 
“breaks through his own plastic essence, like a being who opens the window on which its 
own visage was already taking form,” and that his presence consists “of divesting himself 
of the form which does already manifest him.” Ultimately, “The epiphany of a face is visi- 
tation.” (Lévinas 1996: 59) For Lévinas, this is about the phenomenon that constitutes the 
appearance of the Other, which is also a face; as such, it is a phenomenology of the human 
appearance, in which the face is not a pure, optical given, but a something that I cannot 
access, that I cannot grasp because it confronts me, because it was already there and has 
passed and, in the process, has left behind a trace. The Other, with his face, is not simply 
present in my field of experience, but is experienced as an Alien without being determined 
or influenced by me, because this Other and Alien has always preempted me, because his 
trace has always taken form already. 

However, as Bernhard Waldenfels (2000: 298) demonstrates in his lectures on the 
phenomenology of the body, this trace does not mean that “everything is out in the open; 
yet neither does it mean that something is there but hidden, which I would only find out 
about through analogy or other intellectual considerations. Rather, it means: there are clues 
of something there in its absence.” Waldenfels demonstrates that we do not get acquainted 
with a face “by registering individual empirical data points so as to assemble a something, an 
x, a face,” but instead the details of the face “are preceded by a physiognomy in which specific 
aspects such as expressions, liveliness, and ways of looking abruptly come to the fore.” (Ibid: 
243) The face distinguishes itself, sets itself apart, asserts itself, becomes an entity in whose 


line of sight I move; I am entering someone’s gaze. Therefore, in a broader sense, the face is 
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in a special way unapproachable and inviolable, whereas in a narrower sense it is grasped as 
“something visible in the world, a form with specific attributes that lead to that face being 
recognized and thus identified.” (Ibid: 391) 

In an even broader sense of the face’s unapproachability and inviolability, according to 
Lévinas (2007: 216), the face of the Other also says: “You shall not commit murder.” Hence, 
the face or “countenance” of the Other —to emphasize its original numinosity— issues an 
ethical resistance; it penetrates my capabilities and is an addition to the inevitable stagnation 
of appearance, and thus it reaches the edges of my capabilities. My own possibilities are 
called into question by the Other. In 4 Thousand Plateaus, Gilles Deleuze and Félix Guat- 
tari (1987: 167-91, 170-71) distance themselves from ethical imperatives such as these—in 
inter-bodily lived worlds from face to face—and, in their rhizomatic thinking, declare the 
face in the narrower and broader sense to be a “bunker-face” (visage-bunker), a surface, map, 
long, rectangular, triangular, with lines and wrinkles, a strange system, “horrible and mag- 
nificent,” amidst the abstract mechanical assemblages of power (pouvoir). 

Within their semiotics, the face is a white wall—significance—and a black 
hole— subjectification —vacant as a projection surface, filling up as a locus of resonance by way 
of an individual unconscious. “The face is Christ.” Year Zero, the creation of the face. “The 
face is the typical European.” (Ibid: 176) The face is also hand, chest, stomach, penis and 
vagina, thigh, leg, and foot. Between capitalism and schizophrenia, the face cannot be sep- 
arated from fetishism or erotomania, it is re-coded: as the inhuman in the human, as the 
close-up, as the lunar landscape, with its uninhabited white surface and gleaming black 
holes, in vacancy and barrenness, towards the disciplining of bodies, from organic layers to 
layers of significance structures and subjectification procedures, on a surface riddled with 
holes, in despotic and authoritarian assemblages of imperialism: a face, what horror (Ibid: 
190). To mix semiotics, the face because a mask that hides nothing but elevates everything, 
is no longer bound to a body, an available subject, a prior signifier— because the face is the 
mask itself. 

Moreover, the face is politics. According to Deleuze and Guattari, the face must be 
escaped: through becoming-clandestine (Ibid: 186) in realms of A-significance and A-subjec- 
tivity, in deterritorialization in real life and in art, as a person slips into a face even before 
possessing one, in bi-univocal relationships with the face of the Other, in elementary units, 
as “a man or a woman, a rich person or a poor one, an adult or a child, a leader or a subject, 
“an x or a y.” (Ibid: 177) Beyond ontology or existence, the human body gains face in becom- 
ing, within the immanent plane, which also includes the image: a close-up is a face and “the 
affection-image is the close-up,” as Deleuze (1986: 87) explains in Cinema 1. After all, the 
close-up can be treated like a face (visage), it is viewed in the face (envisagee), and it obtains a 
face (dévisage). As a reflective surface and an intensive micro-movement, it is a face in and of 
itself. It is situated between perception and action, contains the inherent power to separate 
the image from space and time in order to demonstrate pure affect in expression, in acts of 
de-framing (décadrage), from process to passion, as with the actor Maria Falconetti playing 
Joan of Arc in Carl Theodor Dreyer’s 1928 silent film. 

The film isolates the facial expressions and the physiognomy of the visible human 
being (Bälasz 2001: 48-49) without placing them in context, without a counterpart; they 
function as the “white wall” and “black hole,” empty projection surface and locus of reso- 
nance to be filled, carrier of affect. The film actor’s face is also a face-object, as Roland Bar- 
thes (1972: 56) writes about the face of Greta Garbo, which is worthy of admiration; and 
with its makeup, with its “snowy thickness,” it equates, in turn, to a mask: not painted, but 
plaster-like, closed-off, ephemeral, compact. Garbo’s face is a temptation to a fofa/ mask, as 
the archetype of a human face, out of which only the eyes tremble, vulnerable and wounded, 
“black like strange soft flesh, but not in the least expressive.” (Ibid.) Yet specifically, in 
the film Barthes is speaking of, Queen Christina,’ the face does receive a counterpart, in 
the interplay with antagonists, in the shot/counter-shot technique, in conversation, conflict 


(Auseinander-Setzung, literally “taking apart”), it becomes the carrier of the gaze, which is 


more than a written face, a vacant surface, than white paper and black inscription (Barthes 
1981: 122): Garbo as an actor (in the social theory sense) risks her exterior, she becomes, 
paraphrasing Susanne Valerie, the vivid venue of the fragile exposure of a human being who 
seeks a mask in her appearance — in front of the eyes and ears of Others: observation, reflec- 
tion and control as well as nature, sensibility and passion allow the voice to rise, the words 
to pause, the sounds to vary, and the limbs to move. Thinking and experience construct 
and structure the “laboratory of existence” whose existence has always been a human one, 
in which the actor can become a “study subject,” in part via her faciality and corporeality, as a 


carrier of existence, affect, and gaze, as an eccentric face-subject (Granzer 2011: 96). 
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A BRIEF 
AVATAR- 
NETWORK 
THEORY 


DIETER MERSCH 


ACTOR-NETWORKS AND 
OTHER OBJECT-ORIENTED 
ONTOLOGIES 


For more than two decades, Bruno Latour’s actor-network theory has dominated discourse 
in Germanophone cultural and media studies (Latour 1999, 2005b: 21-26, 46-50, 79-82). 
Originally conceived for sociologically-informed research on science, particularly in the field 
of science and technology studies, actor-network theory engages with scientific research 
and its historical changes by analyzing networks of actors. Actor-network theory defines 
the category of actors in a broad sense to include scientific objects, experimental systems, 
theories, instruments, methods of measurement, archives, scientists, and their assistants. 
It focuses not so much on the theory of science, which explores how scientific objects are 
constituted and the application and genesis of scientific theories. Rather, it is more of an 
ethnography of the sciences, of how they organize their practices and establish global net- 
works. Actor-network theory seeks to uncover a// “agents” that have a part in the scientific 
process. Instead of restricting itself to intentional action, the theory understands “agents” in 
the weak sense of “co-actors” situated in a field of forces. For actor-network theory, scientific 
praxis is rooted not in an ensemble of ideas and intentions but is a function of a web of 
relations that provide the foundation for what is called “objectivity” in different historical 
periods (Daston Galison 2007). In short, the approach aims to critique the fantasy that 
science is a wholly transparent, controllable process and not a complex field of diverse, some- 


times contradictory elements and their relations, which only bring forth that which is to 


be defined as a scientific object when they come together. ‘The basic claim underlying this 
critique is that facts are always forged out of and connected to a series of factors that can 
never be identified in their entirety. ‘The result is that scientific constructs remain resistant 
and contain blind spots that relativize their validity. Accordingly, the “networks” (réseaux) 
made up of connections, objects, technologies, and people precede that which constitutes 
science, its facts, and its discourses. 

The success of actor-network theory and its transfer from science and technology 
studies to cultural studies, media studies, and aesthetics and art has been remarkable. Having 
garnered the status of its own paradigm, it has established itself as one of the key theoretical 
approaches, and this despite the fact that alternative philosophies of praxis—one prime 
example being that of “artistic research” — gained an upper hand around the same time. 
‘The transfer to other disciplines was not clear-cut, though. Distortions, misunderstandings, 
and exaggerations were all part of the process, particularly as concerns the theory’s main 
category, the “actor-network.” The expression is one word: there are not, on the one hand, 
networks and, on the other, actors or agents as separate entities that then fuse together 
into a conglomeration. Rather, the term signifies the heterogeneity of actor-relations itself, 
with the “nodes” represented by subjects or objects and the “edges” represented by relations 
that have their own individual affordances and power to act. Thus, we are not dealing with 
neutral units, but with closely interwoven “actants.” Among them are things in particular 
arrangements or concepts in the context of discourses, materials, and experimental practices 
and settings, all of which only attain their “effectiveness” in their mutual interactions. They 
are all inseparably bound up with one another, and it is the unique relations that generate 
divergent effects that can rarely be reconstructed identically and thus must be analyzed 
as a whole. Thus, actor-network theory is a type of relativism that demands that we think 
holistically and place a whole assemblage of mutually interacting elements in relation to one 
another. These elements’ constellations mark a break with classical modes of description 
because the purpose is not to identify causality, but to deduce “scripts” out of interdepen- 
dencies and connections. 

However, standing in contrast to its application in the history of science, one of the 
common vulgarizations of the theory in cultural and media studies and research on artistic 
practice is to shred the network of relationalities and extract actors out of it as self-standing 
entities. The term is then often turned into “actors-network theory,” where the silent inser- 
tion of the “s” underscores the mistakenness of the approach. Because while the expres- 
sion “actor-network” emphasizes the network, the “actors-network” emphasizes the actors, 
which, for their part, advance to equally acting agents in a mixture of human and non- 
human elements. This underscores their individual autonomy and frees things and technol- 
ogies from the yoke of anthropocentric paternalism. And it grants objects an almost magic 
power, as if they were living beings vested with their own powers of attraction and some- 
times even intentions, thus systematically blurring the line between the human and non- 
human. Latour himself suggests this reading when he speaks of an “object’s activity,” even if 
in the same breath he says that objects can, at most, only be “mediators” or “intermediaries,” 
explaining that “specific tricks have to be invented to make them talk, that is ... to produce 
scripts of what they are making others—humans and non-humans—do.” (Latour 2005b: 
79) This is particularly true of autonomous technological things and their assemblages, and 
especially objects with artificial intelligence that are seemingly capable of making their own 
“decisions.” Examples include self-driving cars, diagnostic expert-based systems, robots, 
communication assistants, and avatars. But these artefacts do not act “autonomously,” at 
least not in anything more than a metaphorical sense, because they always act within the 
predefined framework of a program and probability functions. 

In philosophy, the theory has been further developed and radicalized by the New 
Materialism and object-oriented ontology. Especially prominent here is the work of 
Graham Harman, whose “speculative realism” has had spectacular success at schools of 
art in particular (Harman 2007: 171-206; 2018): “Something important is happening. 
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In our profession, there have never been better times to be young.” (Bryant, Srnicek, 
Harman 2011: 1) Harman goes well beyond Latour and asserts—in an anti-Cartesian 
state of “naivete” (Harman 2011: 5)— that there is a principle equality among all entities, 
whether it be humans, animals, viscous substances, atomic particles, or, as Harman adds 
somewhat flippantly: “diamonds,” “rope,” “neutrons,” “pens,” “eyeglasses,” “tools,” “beating 
hearts,” “fire,” “Egypt,” “armies,” and “football stadiums.” (Ibid., 5-7, 20, 37, 142 passim) 


He writes: “Everything both inside and outside the mind is an object,” (ibid.: 143) objects 


» 


that maintain the most diverse relations with one another, most of which, however, we 
cannot observe. The picture of the world painted here is similar to that of a giant living 
organism in the mode of vitalism, but with the significant difference that its reasoning is 
purely speculative. For Harman, it is apparently wholly indifferent if two balls hit each 
other or if a person hits another person. This denies every difference, every hierarchy, every 
distinction between actio and re-actio, because in principle, things, plants, and other entities 
have the same rights as the people who make judgments about them. The main target of 
critique is what Quentin Meillassoux calls “correlationism,” (Meillassoux 2011) to which 
he ascribes the claim that everything that we can know or experience of the world is always 
already shaped by our thinking, concepts, judgments, and categories and is thus subjectively 
determined. In contrast—and in distinction to actor-network theory, which would not dis- 
pute this claim — speculative realism, of which object-oriented ontology is a species, asserts 
that there is an “ab-solute” reality independent of humans in which all events are placed on 
equal footing, whether they simply affect humans or whether they occur “between” humans 
or “between” objects. This confronts us with two givens: the objects, on the one hand, and 
networks of relations, on the other. The “humanist” uniqueness of humans is consequently 
erased. ‘The only thing that matters are events that, regardless of their ontological origins, 
have relations with one another. ‘The philosophy rules out the possibility of privileging any 
particular perspective. On the whole, the approach is inconsistent, because it is only possible 
to make this claim by simultaneously reinstituting the primacy of the human thinking about 


these issues, thus surreptitiously reviving nothing other than the old anthropocentrism.' 


POST-HUMANIST FANTASIES 


Anti-humanism or post-humanism in this form, whether it be vulgarized actor-network 
theory or object-oriented ontology, clearly remains a type of humanism. But the following 
will test its plausibility by analyzing a special example: the avatar, or more precisely, avatar- 
human constellations. Alongside cyborgs, robots, digital characters, and other seemingly 
“autonomous” technological constructs, much discourse in media and cultural studies is all 
too ready to grant agency to avatars—as purely simulated beings that have no other exis- 
tence than as mathematical models. This, it is alleged, dethrones the hegemony of human- 
ism. However, the claim is implausible, because the artefacts themselves have a humanist 
origin. After all, as Martin Heidegger (1977) writes, technology is only the tip of the ice- 
berg of the metaphysical disposition that he criticizes in his “Letter on Humanism” as a 
human fantasy of power. But the claim serves to multiply the number of such agents and 
counterpose the primacy of human activities with more “autonomous beings.” The point is 
to relativize the precedence of humans by showing that they have to deal with heterological 
agents that might even be better than them in certain respects. Many theories of art have 
gone beyond this technological post-humanism by ascribing agency to every material, thing, 
medium, or technical apparatus; however, these efforts often remove the objects from their 
context and do not reposition them in a network of relations. Props, paints, slides, special 
effects, light, computer animation, and animated figures suddenly gain the capacity to for- 
mulate intentions and intervene in and shake up the action—just like smart phones and 


other programmable smart things evoke emotions that can bind people to them. 


Such fantasies were once the exclusive object of theories of schizophrenia. If one reads them 
as evidence of objects’ “capacity to act,” the question arises as to how one defines “capacity” 
and “action” in this context. Certainly, all kinds of objects can influence us and our envi- 
ronment, but this was not even controversial for classic theories of causality. The thing is 
causa of an effect, whether it be as the initial spark of a chain reaction or as hindrance, as 
an Odradek that always pops up where it does not belong or, conversely, is noticed in its 
absence as a “tool.” As absentia, it has a further effect, because we experience the significance 
of a thing more gravely the more we miss it. Ascribing “agency” in the sense of a “capacity 
to act,” however, goes well beyond this, because it concerns autonomy, intentionality, and 
thus independence. Certainly, things, other living beings, and “nature” affect us and some- 
times make us act or change our plans, such as when they stand in the way and counteract 
our intentions. But they only do this because we had plans and then only by negating those 
plans. At other times, they require from us sophisticated techniques of diagnosis, methods 
of making visible and processing. Indeed, their vulnerability, fragility, and finitude are inte- 
grated into technology itself. They generate breaks and “force” us to behave in certain ways 
or to repair and revise. But the subtle difference between this conceptual insight and the 
various “actors” theories lies in the fact that the former works with terms like re-action, 
change, temporality, and negation, which are all associated with contingency, decay, and resis- 
tance. Thus, essential here is not a—positively connotated— “capacity to act,” but, at the 
most, negativity. 

But resistance does not necessarily constitute capacity, which is only true when the 
resistance is wanted and conflicts with the rules in protest. A tree blocking a road, a dam 
diverting water, a turnstile, an opaque structure that cannot be understood, and mutating 
viruses—none “want” anything or refuse to respond. Rather, they condition our actions such 
that we a/ways already have an essential part in these events. This also means that we cannot 
subtract our role from Zheir alleged actions. At the same time, the word “compulsion” says too 
much, because the issue is not how things and objects obstruct our actions, but is simply a 
matter of formal necessities. The discursive confusion lies in the fact that “actor” is a preten- 
tious word that commits a category error when it is taken literally to suggest an activity that 
cannot be distinguished from passivity. What is more, passivity comes “earlier” than activ- 
ity, which makes the “actors” perspective generally deficient, as will be shown in more detail. 

‘The adequacy of such ascriptions is another issue, particularly as concerns the modality 
of relationships that we enter into with others as well as with that which we are not. This 
includes technological artefacts and artificial beings that we ourselves have created in order 
to busy ourselves with their automatic activity. They only seem to approximate us, touch 
us emotionally, subvert our affects and senses, provoke us, or engage in dialogue with us. 
The (ultimately cybernetic) theories of cultural and media studies as well as theories of the 
new “ecologisms” (Morton 2016) and parts of artistic practice consist in the notion that 
interactive technological media will enable us to overcome humans’ monopoly on action, 
because current and—even more—future technologies will compel us to accept the desta- 
bilization of our dominance by developing their own affects, reading our emotions, inter- 
vening in our actions, applying our wishes, making decisions distinct from those made by 
humans, and thus being equal “playing partners.” (Hayles 1999) But is it possible to call 
this “cooperation”? Do technologies really enter into “partnerships” with us in the genuine 
sense of the word when they interact with us? And are the “decisions” they make more than 
just algorithmic, randomized re-actions? These questions are particularly important for the 
many avatars that are conceived as simulative representatives of ourselves or other, poten- 
tially malevolent characters that serve as assistants, advisers, fictional bearers of identity, and 
much more. The following thus discusses whether we are justified in speaking of “power,” 
“cooperation,” “partnership,” “foreignness,” and “malevolence” when talking about such 
artificial beings, or whether these are not just concealed anthropomorphisms that fly in the 
face of the emphatic claims made by object-oriented ontology and actors-network theories, 


thus revealing that they revive the “humanism” that they themselves seek to fight against. 
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FIG. 1 


RELATIONALITY VERSUS 
RELATABILITY 


The crux of our critique lies in the reality of “social” relationships and questions about their 
reduction to networks and formal ties. The comparison between avatars and interhuman 
relationships underscores the intensity of this issue. First, it should be recalled that network 
theories are rooted in mathematical graph theories that only know two types of entities: 
“nodes” substitute objects, which have no ontological predetermination and thus can be 
treated as empty containers that can hold anything, while “edges” represent the connecting 
lines and can stand for all types of relations. They, too, are “empty spaces,” determining 
in advance only the number of connections, but not their modality. Mathematical models 
abstract from their content, so that it appears irrelevant if the “edges” connect humans or 
non-humans. After all, crucial is the structure and its qualities, not experiences like social 
trust, communicative reciprocity, and interhuman interactions. Moreover, are sociality, 
cooperation, and understanding—in short, the specific connections between humans and 
their own capacities that were once associated with concepts like koinonia and religio—at 
all compatible with avatars? Can “living 
together” in the sense of political “par- 
ticipation” be compared with the mere 
sharing of a common space between us 
and technological and other artefacts 
and images, artificial beings and simu- 
lacra? What does it mean to “encoun- 
ter” an avatar— perhaps even an avatar 
of myself as at once a sort of alienating 
self-encounter? What does it mean to 
look at “them”—or, better, “it”—in 
the eyes, to be affected by “them,” to 
respond to “them”? And how should 
we view two avatars’ or robots’ “love” 
or “sexual attraction” to one another? 

Should we feel shocked, 

disgusted, amused, tricked, 


excluded, entertained, or 


CHRIS CUNNINGHAM, MUSIC VIDEO FOR pettya, Caney, On! 
BJÖRK, “ALL IS FULL OF LOVE” (1999). own inventions, be treated 


as equals and granted the 
same rights? Or must we 
distinguish between “rela- 
tionships,” which are always constituted by a “pull,” (Mersch 2010) and the diagrams of 
a network? In other words, do connections —and especially technological connections— not 
always constitute a deficient mode of relating that has interhuman sociality as a precondi- 
tion? Must we not have already entered into social “relationships” in order to understand 
them? 

As far as these questions are concerned, one might say that avatars serve as extreme 
tests of what it means to act autonomously, be an actor, and develop relationships with 
non-human artefacts. The diagnosis offered by object-oriented ontology seems clear: we have 
no privileges over artificial entities, which are not just simple passive objects. Nothing gives 
us privilege, greater status, or a sense of superiority over them. And yet, the assertion that 
there is equality between us and them involves a distortion, because the problem of fictive 


interactions with avatars is not the alternative between hierarchy and equality. Rather, the 


decisive issue is the basic incommensurability between these relations themselves. This is to 
say that human relationships and artificial relations are incomparable, because the former 
signify an event within a complex social order that is defined by alterity, while the latter 
just derive from a network of positions that must always be embedded in social systems but 
cannot be larger or more expansive than them. Certainly, one can argue that ecologies pre- 
cede economies and that environmental totalities encompass social totalities; that sociality 
is nothing without the “nature” that makes it possible. However, ecologies are always either 
described through graphic, mathematical structures, which remain neutral, or their seman- 
tics —and thus their anthropomorphic quality—are already a derivative of the social world. 
There is no ecology, no nature from which humans and their sociality can be removed, 
because this sociality always precedes the analysis of nature. ‘The defense of ecology, too, is 
a social task that applies the experiences of the social to environment and nature, because 
they are a precondition for our living together. The philosophical weight of these reflections 
consists in the insight that the social is even constitutive for our ability to distinguish between 
objects and humans at all and for the fact that, where this distinction is missing, the sense 
of the genuine sociality of the human disappears. Without habitual social relations, we could 
not only not interact with one another, but would also not be able to interact with things 
and thus avatars. Indeed, we would not be able to relate to anything at all, not to speak of 
thinking, understanding, telling stories, creating art, and doing science. One of the possible 
consequences of this critique is thus that equalizing the ontological will lead to manifest 
sociopathologies that destroy the difference between relationship or relatability and relation. 

Versions of actor-network theory prominent in media and cultural studies and art 
theory evidence this dissolution of difference, because they subtract precisely those aspects 
of the approach that are interesting for social science. With respect to research praxis, 
Latour’s actor-network theory is a social philosophy. Its brilliance consists in conceiving 
of actors as network figurations in which human relationships are always active. In contrast, 
post-humanist appropriations of the theory only function by abstracting from social reality. 
While Latour sought to delimit the social, the post-humanist version of ANT completely 
erases it. In defense of his own work against some of his followers and critics, Latour wrote 
an article in 1996 titled On Actor-Network Theory: A Few Clarifications plus More than a Few 
Complications in which he clarified that the postmodern appropriations of ANT were misled 
because “actor-networks” should be understood as an “ontology” with a genuinely “networky 
character” and not as a conglomeration of ontological things (Latour 2017). Actor-net- 
works are active interconnections, not interconnections of “actives.” The widespread talk of 
the agency of things that allegedly necessitates a redefinition of the relation between subjec- 
tivity, materiality, and alterity, thus rests on a systematic false reading. The question of who 
or what the “actors” are is incorrectly formulated, because there are only connections and 
co-constitutions, whereby the prefixes “co-”and “con-” are primordial. And every network 
has its own shape that must be reconstructed. Thus, no avatars exist as isolated artificial 
identities that, in the guise of “simulated subjects,” undermine “humanism,” just as sur- 
prising “encounters” with them are unthinkable. Rather, there are only “avatar-networks” 
comprised of an ensemble of computer scientists, software engineers, displays, settings, 
discourses, visualizations, data flows, internet connections, online communities, narra- 
tives, fellow players, and much more. These actor-networks are always already shaped by 
human structures, one reason being that they are human technologies. In other words, we 
“encounter” avatars at most in play settings, films, and virtual realities in which we already 
anticipate their presence. 

Thus, Erich Schüttpelz (2016: 235) correctly claims that actor-network theory avoids 
“fixing artefacts, social organizations, and signs” as well as persons and things in favor of 
delineating foundational orders of relations and their processes. From this perspective, 
agency is a function of “chains of operation” that should be characterized as “insistent trans- 
actions” and not as a function of autonomous decisions (ibid.: 237-238). As a consequence, 


we are confronted with the “postulate of the primacy of social and technological chains of 
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operation over the entities involved.” (Ibid.: 237) Agency should thus not be attributed to 
individual actors, but only to the complexity of a relational arrangement. This also holds for 
avatars, which are nothing but aggregates and not “living” humanoid beings that we could 
enter into dialogue with on equal footing. Instead, their agency is constituted through at 
least three factors: first, programs of similarity that are gleaned from social processes, even 
if artificial intelligence drives them; second, the quality of being a technological artefact, 
which enable the adaptation of certain movements or reactions with the help of tools like 
motion-capture; and third, signs, which include data, drawings, and archives, so that they 
can only be understood and treated as objects of a relationship when they are viewed within 
the whole constellation to which they belong: “Actor-network theory—despite rumors that 
say otherwise—does not grant the persons, things, artefacts, natural beings, instruments, 
media, writings, images, and data centers that it describes a power to act or ‘agency’ of same 
strength, equal quality, same source, not even same status. Rather, all that actor-network 
theory demands is the equal consideration of the ‘agency’ — the power to act— of artefacts, 
signs, persons.” (Ibid.: 242) 

The specific charm of the theory, which has had such an enduring effect on cultural, 
media, and art studies, thus derives from how it brings together the whole heterogeneous 
ensemble of relevant aspects that require an implicit “ontography” to be described. Their 
relational scripture, which has much more in common with Derrida’s theory of writing than 
most proponents of actor-network theory are aware, thus excludes every attempt at ontol- 
ogization or the conception of agency as a quality of objects. Instead, as Schüttpelz writes, 
agency unfolds in an “operative process,” (ibid.: 243) which is to say that it comes out of a 
processual co-relationality that is similar to a “graph” of forces like in theories of physical 
force fields. In this sense, the non-human elements are more akin to the pathic than to poiesis. 
Applied to avatars, this means that they are hybrids in which relations and their relata are 
ordered together, but in such a way that the latter only obtain their specificity through the 
former. They are heterogeneous associations in which the copu/a is the most important part, 
such that the connections mobilize an entire apparatus that governs interactions with them, 
their possibilities, and their limits—in short, their operability. “One difficulty certainly 
concerns vocabulary. The equation of actors with the relations their actions place them in 
and their power to act, the equation of ‘actors’ with ‘actor-networks, the hyphen between 
actor and network, and the fact that every ‘network’ (every action relation) can appear as an 
actor that enters into other networks— all of this promotes an activist view of actor-network 
theory.” (Ibid.) 

These points underscore that the human element cannot be removed from the net- 
work. The purpose of actor-network theory is thus the assessment of often forgotten non- 
human elements in the sciences, particularly ethnography, cultural philosophy, and social 
theory. Latour (1993) thus explicitly talks about a “symmetrical anthropology” that would 
remain anthropology and not get lost in the inconsistencies of object-oriented ontology. 
At the same time, this anthropology’s claim to “symmetry” reveals the necessity of the 
interdependency of its elements. Just as no human, social, or cultural facts can exist without 
relations to non-human things that they will have always already transformed, so too are 
there no non-human facts that have not already been influenced by human actions. Thus, 
the “human element” is just as dependent on things as the things and technological arte- 
facts are on human and interhuman relations. Alongside Latour’s many examples, the pistol 
makes this point especially well. The pistol is a piece of technology, but it is nothing without 
the person who holds it and uses it for good or evil. One cannot call it a neutral object that 
freely stands to human disposition, nor can one call it a thing that is itself an actor. One can 
only speak of various human-weapon relations that generate a specific type of violence and 
are violent in every constellation. The issue is not to decide to engage in a violent act, but 
to understand the different modalities of violence that are produced by different relations. 
Thus, we are not dealing with innocent, isolated things nor with individual humans who 


only activate these things through their practices. Rather, we are dealing with impenetrable 


“knots” of various actions and potentials to act. Actor-network theory, with its emphasis 


on chains of operation, thus contains a moment of opacity and contingency, which is what 


makes it so powerful for the analysis of both the sciences and of art and society. 


AVATAR-NETWORKS 


The implication is that avatars only have being at all where networks are already laid out and 


mapped. Avatars never appear and cause trouble on their own, but only do so in the context 


of human-avatar associations with their technological, semiotic, and computer foundations, 


which include all the parameters of simulations and the scenes that correspond with them. 


They parasitically feed on the primacy of social relations that they rely on and from which 


they have “learned” through machine learning. The fascination that we have for animated 
y g 8 


figures and quasi-living dolls carries a trace of this. They embody a vitality that we know 


is only staged, but that captures our attention precisely because it evidences a similarity 


shot through with dissimilarity and 
thus forges a distance to our everyday 
experience. In other words, having 
an encounter with an avatar implies 
already being wrapped up in an ava- 
tar-network that remains subordinate 
to the sociality of interhuman rela- 
tionships. As a result, unexpected 
face-to-face encounters with avatars 
seem impossible because, unlike with 
ghosts and other apparitions, avatars 
never appear as autonomous agents. 
After all, attributing agency to them 
would mean extracting them from 
their networks and elevating 

their status to one of autono- 


mous beings. But in the end, FIG. 2 


they are predictable, even AVATAR: THE WAY OF WATER (2022) 


if they simulate unpredict- 
ability. This is why— like 
with robots—fear is not 


the dominant emotion in our 
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experience of avatars, an insight that stands contrary to the “uncanny valley” hypothesis. 
p 8 y y y hyp 


Rather, dominant are neutrality, contempt, and distance, because, as both neuroscientific 


experiments and experiences with cinema show, 
there is always a certain degree of familiarity that 


we can easily decipher and enjoy. 


FIG. 3 


3-D AVATAR MODEL 
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Thus, the foreignness of the avatar is a myth, just like the fear of humanoid aliens that 
conspire against us and take over in the world of science fiction. It is a myth because they 
are part of the history of technology, which witnessed varying prototypes and degrees of 
perfection before human attributes were directly programmed into them and we began to 
integrate them into our perception. Thus, avatars always originate in an analogy, which 
means that they will never attain the status of alterity that would be necessary to recognize 
them as other others, neighbors, or equal cooperation partners. They are more like mirrors 
without enigma because we know how they are made and what they really are: mathematical 
simulacra. 

‘Thus, despite all of its theoretical productivity, there is something lacking in actor- 
network theory, because the primacy of networks does not allow for a distinction between 
relationality and relatability. Like the graph theories that stand at the foundation of network 
theories, relations denote formal classifications that distinguish only between “nodes” and 
“edges,” but do not necessarily differentiate between the modes of classification that connect 
the networked elements to one another in varying ways. In short, as avatar-networks demon- 
strate, the deficiencies of actor-network theories lie in their Jack of an adequate concept of 
modality that would relativize the concept of relation and, above all, enable a separation between 
relationship and relation, which is crucial for all forms of sociality. Cooperation, neighborli- 
ness, friendship, and dialogue can hardly be sufficiently described with the neutral concept 
of relation, and they do not adhere to the formalism of mathematical functions. This is 
why interhuman relationships are different than human-machine or human-avatar relations 
and relations between things, necessitating that a fundamental asymmetry be charted in 
the symmetry of networks from the very beginning. This is likely a reason why human- 
avatar relations, particularly in games, always have such a stereotypical form and why fight 
ing—the quintessential form of binary relations—is dominant in them. At the same time, 
it helps explain why these rela- 
tions rarely deal with political 
conflict, in which diplomacy 
and its nuances are of utmost 
importance, or love relation- 
ships, which traverse pretty 
much all emotional registers. 

Another way of express- 
ing this is that actor-networks 
and their relational diagrams 
are similar to homogeneous 
webs that are wholly deter- 
mined by the logic of their 
graphs, while social structures 
constitute asymmetric “fields” 
in which identities, belonging, 

and the elements that 


forge or break commu- 


FIG. 4 nity are most signifi- 
SCENE FROM MORROWIND: THE ELDER SCROLLS (2017). cant. This brings us 
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TESTS/MORROWIND-REVIEW-1229438/ because from a philo- 


sophical perspective, 
the reductive versions 
of actor-network theory 
cast doubt on nothing less than the primordial sociality in theories of agency and replace 


sociality with a primitive copy. While they seek to revive social philosophy by proclaim- 


m ing the symmetrical equality of all relations, they are faced with the “ab-solute” difference 
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that follows out of all other relational differences. Responsible for this difference is, beyond 
Derrida, not the “scripturality of writing,” but the never-ending drama of interhuman rela- 
tionships. It is these relationships that produce differences between humans, technological 
artefacts, objects, and artificial beings, such that where these differences are neutralized, 
the sense for the social and thus for genuine human sociality gets lost. Is this not why Chris 
Cunningham’s video for Björk’s song “All is Full of Love” (1999) is so unsettling? And, 
conversely, is it not symptomatic that the pathologies of the social can be perceived in the 
mythical fantasy worlds of computer games and animated movies, whose archaic narra- 
tives are — not for nothing— defined by violence and heroism, as if they were just directly 
channeling the energy of the social? The reason for this shallowness probably consists in the 


fact that the theater of algorithmic models without genuine decisions is not able to do much 


more or much else. 
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PUPPETS, PETS OR 
DEPUTIES. ON THE 
RELATIONSHIP 
BETWEEN ACTORS 
AND AVATARS 


GUNTER LOSEL 


INTRODUCTION 


The practical part of the research “Actor and Avatar” explored the shifting relationship 
between actors and avatars, between self and other, that comes into focus when avatars 
enter the scene. One approach to this wide field we chose was from the perspective of act- 
ing, assuming that the expertise of stage and camera actors would help to describe new 
phenomena of identification or alienation with avatars. In this article, we sketch out the 
theoretical framework exploring the similarities and differences between acting in a tradi- 
tional sense versus acting through an avatar and describe our experimental design of face- 
to-face encounter with an avatar. 

The core questions remained: Can we consider “avatarization” as a different, possibly 
new kind of acting? Possibly one which could be described as digital puppeteering? Or as a 
different kind of mask work? Is it a distinguishable character or role-play, for which theat- 
rical concepts help us to understand an uncommon form of “embodiment without a body”? 
Can we understand the avatar as an extension of the body, something like a prosthesis? Or 
should we consider this form and appearance as something completely new, not comparable 
to anything we are familiar with as actors and performers? 

Avatars are a new phenomenon in theatre in two ways. The first concerns the appear- 
ance of avatars on the digital stage. While the internet has been referred to as a “virtual 
stage” for quite some time, new technologies make this metaphor more striking than before, 
allowing players and users to enter the virtual stage in real-time with a virtual body that 
reacts to the body of the player/user— much like an actor enters the theatre stage. The sec- 
ond concerns the appearance of avatars on the ¢heatre stage, including real-time interaction 
with other characters and the audience. This is a new technical possibility, which creates a 
considerable change from the common use of video and projection on stages. Meanwhile it 
is possible to give the avatar the appearance of liveness and presence. An avatar can react to 


both fellow actors and the audience. In “The Tempest” by the Royal Shakespeare Company, 


the character of Ariel appeared as a real-time avatar since 2016.! Although the aesthetic 
possibilities of its use might be limited,’ the proof of the possibility of an “acting” avatar has 


certainly been made. 


WHAT IS AN AVATAR? THE 
TWO-WORD CONCEPT 


There is still no phenomenology of the avatar and no theory of perception of avatars, 
between image and affectation, motion and emotion. Our project entered a new field, 
where the avatar is seen primarily as a virtual surrogate of persons (Pennig et al. 2012: 
59), or where the scientific discussion revolves around criteria of difference between agent 
and avatar (Bailenson et al. 2005), mimesis (Lanier 2001), or questions of interactivity 
and acting in virtual worlds (Bente et al. 2001, Slater et al. 2000, Giinzel 2012). Further, 
comparisons were drawn between avatars and 
tokens, puppets, cartoons or robots, as espe- 
cially in Klevjer (2006). It is not questioned, Hinduistic Concept 
however, if a personal or person-like relation 


; A . . Higher World 
with or perception of avatars is possible at all, icine 
how far it reaches, what experiences of alterity 
anticipate in this relation, or if a basic separa- 
tion persists between humans and avatars. 

We can differentiate between the trae | --------- 
ditional, hinduistic concept and the modern 
o. Lower World 

concept connected to the digital world. Both (created) 


establish an idea that we refer to as 
the “Two-World Concept”, which 


Modern Concept 


means that there are two separate FIG. 1 


worlds with a boundary that canbe THE TWO-WORLDS CONCEPT OF AVATARS 


transgressed only through transfor- 
mation of the phenomenal body. 

More specifically this sets up 
a concept of a “higher” and a “lower” world, thus describing the transgression into the lower 
world as a “descent,” while crossing the border in the other direction would be an “ascent.” 
This hierarchy is the only specification of the relationship of the two worlds and has import- 
ant implications, since the lower world is somehow dependent on the higher world and, 
when in conflict, the higher world will be given priority. For example, when a computer 
gamer feels hungry, the immersion into the world of the game will decline, he or she will 
shift attention to the real? body and consult the refrigerator in order to eat something. 

This leads to a basic analogy to theatre. Just as the player in a computer game is 
entering the digital world of the game through the avatar, the actor is entering the fictional 
world of a theatrical play through his or her character. The hierarchy of the worlds might be 
comparable, both virtual world and fictional world being created worlds that depend on the 
real world. 

It is also worth noting that the modern concept of the avatar puts the human being in 


the place of a deity, creating some confusion of who is in control, as we will elaborate below. 


1 


https://www.youtube.com/ 
watch?v=MMLOEXW6s38 
(10.12.2016) 
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http://arstechnica.com/the- 
multiverse/2016/11/tempest- 
review-real-time-digital-avatar- 
performance-capture/ (09.12.16) 
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We will be using the word “real” 
quite a lot in this essay, though 
being aware that it is dependent 
on philosophical assumptions 
that cannot be discussed in this 
context. In opposition to “virtual” 
or “fictional”, the term here refers 
to a world that can be perceived 
through a multitude of senses, 
can be supported by intersubjec- 
tive exchange, is permanent and 
independent of mental states. 

As Phillip K. Dick put it: “Reality 
is that which, when you stop 
believing in it, doesn’t go away”. 
https://en.wikiquote.org/wiki/ 
Philip_K._Dick (04.03.2017) 
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Rebecca Smith 


ENVIRONMENTAL VIEW 
ON THE AVATAR 


The two-worlds concept leads to the conclusion that the relationship between player/user 
and avatar also includes a relationship with the wor/d of the avatar. The avatar is not only 
a puppet sitting on a shelf looking pretty or scary! It is always part of a fictional or virtual 
environment, and conceptualizing the avatar without this relationship provides only a 


limited view. The aim of an avatar is to enable partici- 


O_o pation in a world that would otherwise not be accessi- 


ble to the player/user. So the avatar must be seen in the 

Zoey17 \ context of the world to which it grants access and the 

idantification? operations that can be performed through the avatar 
<<>> in this world. 

Immersion? As the illustration shows, the relationship gets 

rg A Ly quite complex when we consider the environmental 

World of Zoey17 aspect of the avatar. The way in which an avatar can 

‘ operate in its virtual world is very important, as the 

7. “cursor theory” highlights below. It seems reasonable 

Real Life: to distinguish between two processes: (1) a process of 

mend Or Rede eed SINN identification with the avatar as a digital 

object and; (2) the immersion into the ava- 


FIG. 2 


tar’s world, story, fiction or game. Though 


ENVIRONMENTAL VIEW OF THE AVATAR _ interdependent, these might be seen as two 


analytically separate processes of avatariza- 
tion. What we have called “relationship” is, 
therefore, a varying mixture of identifica- 


tion and immersion. 


In accordance with this idea, the theatre-researcher Ulf Otto speaks of two different func- 


tions of the avatar; an operative function and a symbolic function: 


An avatar therefore is not a scheme or picture, that overlays the appearance of reality, 
but it is a deputy in two senses: On the one hand the avatar is an operative deputy, 
providing impact within the model as a functional element. It gains its appearance 
while being performed as a figurative practice, that can be described as analogous 
to the embodiment of a character through an actor—even when the avatar is not so 
much emerging in the differentiation of actor and audience but in the differentiation 
between human and machine. On the other hand the avatar appears as a symbolic 
deputy of this figurative practice through which it is generated. 

(Otto 2013, 109. Translation G. Lösel) 


So, the processes of identification and immersion rely on the way the player or user bands 
together with the operative and the symbolic function of the avatar. Otto parallels this 
process with acting, placing his main focus on the operative aspect. One can conclude that 
in virtual and fictional worlds it is not so much what we are, but what we do, that connects us with 
the avatar. This establishes the emphasis on the performative side of the process: To inhabit 
an avatar, the player has to acz. Identification (or whatever one might call it) will arise from 
these actions, and the operations that a player performs when first controlling the avatar 
will be crucial for the subsequent relationship. ‘The focus will therefore lie on the first few 
minutes of the encounter with the avatar, when the player/user discovers how to perform the 


avatar by trying out all possible operations. 


THE STAGE-CHARACTER 


AS AN AVATAR 


In theatre, this process would be called “identification with the character,” where the actor 


not only reconstructs and re-enacts the actions of the character within the play, but also 


builds up a spectrum of possible actions and emotions that seem apt for the character, 


though they are not part of the play. Often this is done through in-character improvisa- 


tions, confronting the character/actor with unforeseen 
situations and finding out how he or she would behave 
in these situations. The actor accesses a state of mind 
and body that is different from his or her everyday self. 
This duplication of the actor into a phenomenal body, 
which is his or her own body, and a fictional body, 
which is the body of the character, has been at the very 
heart of acting theories since Denis Diderot’s “On the 
Paradox of Acting” (1770-73). Some of the intense 
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experiences of theatre and acting can be retraced to 
this paradox of being “two in one.” 

Using this model, one might conceive 
the character embodied by an actor as an ava- 
tar that allows him or her to enter the fictional FIG. 3 
world of a play (fig. 3): 

The actor will transform into the char- 
acter by embodying it and— depending on the 
acting school he or she comes from — by identi- 
fying with the character." In the fictional world 


he or she will be able to interact with other characters, 


in similar ways to how they would interact in the real Is 


world, using voice and body to convey meaning. At the 
same time, the actor stays in co-presence with the other 
actors and the audience in the real world. Transferring 
the visualization to the avatar, the relationship looks 
similar (fig. 4). 

The principal difference is that there is no 
co-presence in the real world. The players (or users) will 
not share the same time, space or focus of attention. 
Even if they are in the same room at the same time, 
there will be no real-time interaction between 
them. When it comes to interaction in cyber- 
space, the main feature is not co-presence FIG.4 
but participation, because, as Otto (2013: 99) 
points out, presence in the computer world is 
not “just there”, it has to be actively generated 


by participation. A reader of a chat who does 
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ROLE PLAYING IN THEATRE 
(SOURCE: LOSEL ON THE BASE OF HORBELT 2001) 
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not add a comment, is not present for the other participants, while an audience member in 


theatre is present even when he or she is similarly passive. This main difference, then, is that 


interaction in the virtual world is dominated by the choice to be present by participating or 


4 


not to be. This choice leads to another aspect of interacting avatars, the preference of inter- 


ROLE PLAYING IN CYBERSPACE 


Of course, postdramatic and 
modern forms of theater have 


actional partners, which means that we are free to choose. 
In the virtual world there is no embodiment in the conventional sense because there is 


no body. Instead, the player/user can choose an appearance that is far beyond biological and 


questioned the concept of identi- 
fication and character altogether, 
but these forms are neglected 
here, because they have no con- 
nection to the idea of an avatar. 


227 


228 


physical restrictions, which leads to infinite options that are independent of the real body. 
The relationship between the real body and avatar is very loose. Still there are limits as to 
what appearances we can relate, as Pindar Saygin points out (Saygin 2012). How human- 
like does an avatar have to look? How human-like should its behavior be? In any case, there 
is an aspect of representation that defines the relationship between player/user and avatar; 
representation does not necessarily mean identification. In most cases—e.g. dating plat- 
forms—there is probably no identification. Until now, research has focused on the theme 
of choice: Which avatar would people choose to have them represented accordingly? Future 
research should take into account the process of identification. Computer players recount a 
state of identification with their avatar only after hours of playing, so one can assume that 
identification comes with operating the avatar. 

Allin all, encounters through avatars have some similarities to theatre, but the degrees 
of freedom are much bigger, both in choosing one’s own appearance/role and in choosing 
the interaction. The avatar—like the stage-character—is a form of duplication, but in an 
externalized form, transgressing the border into another world much more radically than 
the character stepping into the fictional reality of the play. The avatar is an out-of-body 
duplication, leaving the body of the player or user behind when entering the other world. 


SYMBOLIC VERSUS 
OPERATIVE DEPUTY 


(1) SYMBOLIC FUNCTION 


Until recently, most research has focused on the symbolic function of the avatar and the 
question: Which avatar will best represent me and which one would I choose in order to be 
recognized by friends? Under this premise, individuals will choose an avatar that is similar 
to their own appearance but has a degree— 20-30% — of “avatarization” towards an average 
face. People seem to find themselves best represented when the avatar is similar, but not 
identical. Instead they appreciate their avatar to be around 20% morphed in the direction of 
an average face, a representation that they evaluate as being objectively very similar to their 
own face (Blom et al. 2014) (Epley and Whitchurch 2008). Of course, the avatar is quite 
often not used in order to be recognized, but rather in order to hide in anonymity. This is 
not a representation of the self in the digital world, but rather a disguise, yet still it can be 


summarized under a symbolic function. 


(2) OPERATIVE FUNCTION 


Umberto Eco uses the example of a broomstick that becomes a horse in the frame of play 
(Eco 1977: 209). Obviously, there must be some physical features that make the broomstick 
apt for the symbolic function. It has to have something like a “head”, be long, and some- 
how “hairy” in order to represent a horse. But more importantly, the player must be able to 
handle the prop in certain “horse like” ways, e.g. sitting on it, moving it around in the room, 
jumping and so on. The symbolic function has to match the operative function. Similarly 
one could describe the symbolic function in the terms of J.J. Gibson’s “A ffordance Theory”: 
an affordance is the possibility of an action on an object or environment (Gibson 1979). 
The operative function thus does not rely solely on the symbolic function but can 
be independent. A consequent expression of this thought is the “Cursor Theory” that was 
introduced by James Newman in order to describe the player-character relationship in video 
games (Newman 2002). He rejects a character-based understanding of the avatar. Identi- 
fication with the avatar, in Newman’s view, has little to do with identifying with the visual 


features of an avatar, but with the operational function; agency and control are the main 


qualities of this relationship. It is not what the avatar looks like that makes it possible for 
the player to “inhabit” it, but what the player is able to do through the avatar. In theatrical 
terms, one might speak of a performative relationship. It is only there when acted out; it has 
no ontological status beyond this. In acting there is no parallel to this kind of relationship 
because the stage relies on phenomenal bodies. Drawing on the— quite radical—cursor 


theory, the relationship between player and avatar would not need embodiment at all. 


QUESTION AND METHOD 


In order to find out more about the operational aspect of the process of inhabiting an avatar, 
we set up an artistic experiment. ‘The artistic and experimental partial project relates to the 
concrete picturing model of the comparative situation outlined above. The framing of the 
relation between actor and avatar, which is still very general, was then further reduced to 
perception of facial expression. In this way, voice, physical movement, behavior and other 
bodily actions were excluded from our focus, which therefore relied solely on facial mirror- 
ing. The experiment concentrated on “signif- 
icant” expressions, such as “terror” or “joy”, 
thus reducing the focus further. 

As a result, the specific aesthetic 
arrangement rested in comparing three groups 
of experimentees— professional actors, act- 
ing students and amateurs—and their mir- 
rored expressions recorded in real time. To 
do this, we used real-time face tracking soft- 
ware. In the general sense of an “open circuit” 
of a self-reflexive arrangement, the recorded 
“avatarised” facial features were technically 
“alienated” (see pictures). Simultane- 
ously, the “acted” mimic in the per- 


formers’ mirroring was recorded in FIG.5 


two lines of facial expressions: joyas AVATARIZED FACES OF ACTORS 


expressed by a human and expressed 
by the same person as an avatar. The 
experiment resembles Nam June 
Paiks’ TV-Buddha (1974), although now translated into the realm of computer animation 
and held open by an immediate responsive structure; the actor becomes his own spectator 
and has to learn how to deal with his alienation. The question within the artistic experi- 
mental structure, therefore, lead specifically towards the respective actors/performers, their 


reactions, possible learning, self-training and conditioning. 


RESULTS: METAPHORS FOR 
A NEW RELATIONSHIP 


Metaphors are common in the literature on avatars and several of them derive from the 
field of theatre. The emergence of metaphors can be understood as an attempt to embed a 
new phenomenon into concepts already known, trying to find images that fit the subjective 
experience. In what follows we sketch out these metaphors and discuss their connection to 


acting. 
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PUPPET 


As Otto points out, one of the earliest metaphors in the history of digital agents is the 
puppet. It is mentioned in 1984 in the context of the computer game “Habitat”: 


You were reaching out into this game quite literally through a silver strand. The 
avatar was the incarnation of a deity, the player, in the online world. We liked the 
idea of the puppet master controlling his puppet, but instead of using strings, he was 
using a telephone line. (Otto 2013, 108) 


Referring to marionettes, this metaphor has strong resonance in acting theories, where the 
marionette has been considered as an ideal actor, as in Heinrich von Kleist’s “Uber das 
Marionettentheater” (Kleist 1978, orig. 1810) and Edward Gordon Craig’s concept of the 
“Übermarionette” (Craig 1911). The main argument for the puppet on stage has been the 
absence of emotion and egoism, which, in the eyes of Kleist or Craig, ruin human actors’ 
performances. The avatar might possess the same advantages, conveying more neutrality, 
innocence and less effort than the original actor. If the puppet can be seen as an adequate 
metaphor for the avatar, the limitations of expression must be considered an advantage 
rather than a disadvantage, which means that the more technical progress overcomes these 
restrictions, the less powerful the avatar will be as an actor. So, as an operational function, 
the puppet/avatar lives through its deadness, which evocates projections; the puppet has to 
be clumsy. Otto states that the metaphor of the puppet embraces the idea that the puppet is 
completely under the control of the player, while the avatar seems to somehow possess the 


player or user. Hence the avatar is much more powerful than the puppet. 


MASK 


Another basic theatrical concept that has been applied to avatars is the concept of the mask. 
Ever since, masks “can both conceal and reveal. They are agents of disguise and of disclo- 
sure” (Pitches 2003: 58). This brings up contexts of ritual, trance, and possession as well as 
concepts of theatre like the Commedia dell’Arte. 

Through the mask, a new entity, often thought of as a God, will make its appearance 
and it can be much more powerful than the wearer of the mask. In acting theory, the mask 
has also been a metaphor for the process of being inhabited by the character the actor plays. 
‘The player gives up his or her personality in order to give control to the mask (Johnstone 
1987). So, in this metaphor, the avatar can be thought of as an entity of its own, independent 
of the player or user. It has its own needs, thoughts and intentions, so it can be encoun- 
tered like another person. It is an encounter through transformation. Mask and player don’t 
appear at the same time, but in sequence; when the actor is there, the mask is not and when 
the mask is there, the actor is not. So, the metaphor also implies a complete identification 


with the avatar, but one that is limited by time. 


PROSTHESIS 


Rune Klevjer writes about the avatar in computer games and supports the metaphor of a 


prosthesis of a prosthesis: 


The relationship between the player and the avatar is a prosthetic relationship; 
through a process of learning and habituation, the avatar becomes an extension 
of the player’s own body. Via the interface of screen, speakers and controllers, the 
player incorporates the computer game avatar as second nature, and the avatar 
disciplines the player’s body. 

(Klevjer 2006, 10) 


The avatar is thus considered an extension of the body, inhabited by the self in similar ways 
as the rubber hand is inhabited by the self in the famous experiment (Botvinick and Cohen 
1998). This experiment also demonstrates that the integration of the rubber hand into the 
body-scheme is dependent on sensual stimulation — it is not just a cognitive process but can 
be actively evoked through clever manipulations of the perceptual input. The avatar could 
thus be conceptualized as a full-body prosthesis. There has been some experimentation, if 
the results of the rubber hand illusion can be transferred to full-body experiences (Metzinger 
2015). This seems possible in principle, but the identification appears less intense. It is not 
so easy to trick the self into inhabiting a virtual body, so the metaphor of the avatar as pros- 
thesis is not as strong as it seems at first glance. The avatar is able to operate in a fictional or 
virtual world and is separated from the physical body. Klevjer sees this contradiction within 


the metaphor: 


Which brings us to the final function of the avatar that needs to be pointed out: 
unlike an instrumental extension (a tool), the avatar does not expose our actual 
bodies to the environment; it only exposes itself, as a vicarious body. 

(Klevjer 2006: 96) 


This brings him to the conclusion, that the avatar has a double-nature: An avatar is an exten- 
sion that is also a model (Klevjer 2006: 94). 


MODEL 


A model here is understood as a prop that can operate in another world like a fictional or 
virtual world, for example a toy car, a radio-controlled airplane or a drone. It does not matter 
if they are manipulated by hand or if they move by electrical engines, the main quality is that 
they “... are external in relation to us as participants and can be interacted with as autono- 
mous objects. Their fictional significance emerges from this interaction.” (Klevjer 2006: 77). 
Drawing on Kendall Walton’s theory, Klevjer characterizes the model in fictional worlds as 
a “dynamic, reflexive prop”. It is independent from the body of the player or user but allows 
him or her to enter the other world and to operate in this world, while staying safely in his 
or her own world. The control is usually total; the model has no autonomous life of its own. 
The typical model in this respect would be a radio-controlled model plane, which enables us 


to take to the sky even if we are firmly grounded on the earth. (Ibid: 88) 


GAME CHARACTER, TOKEN, CURSOR 


Game characters can be very abstract, like they are in board games; wooden cubes, beans, 
or coins will do. Their function is almost not representational at all, but mainly operative; 
they allow the player to locate him or herself in the world of the game and perform certain 
operations, but they have no, or almost no, phenomenal body. They can take the form of 
tokens or, in the digital world, the form of a cursor. The cursor theory stands for the most 
abstract concept of an avatar. In the experience of the player/user, one would expect no 


representation at all, only operations. 


ROLE/CHARACTER 


The avatar might also be conceptualized as a kind of role-play, with the representational 
aspect being the costume and make-up while the operational aspect would be the ways a 
character moves and acts within the fictional world. There have definitely been cross-overs 
between acting and digital characters, as Otto points out: There is a growing community 
of cosplayers, who strive to embody fictional or virtual heroes like Lara Croft (Otto 2013). 
Lara Croft also gives an example for a digital character that is later embodied by a real actor 
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(Angelina Jolie), while the voice always had been of real actresses such as Shelley Blond, 
Judith Gibbins, Jonell Elliot, Keeley Hawes and Camilla Luddington. There are also a mul- 
titude of examples of actors being digitalized through movement-capture and face-capture, 
giving life to animated characters on the screen. 

The main difference is the fact that the body of the avatar is not subject to the laws 
of physics and biology; it can take shapes that are biologically impossible, it can fly, it can 
bend, it can transform, it can overcome time and space. Comparable experiments have been 
conducted by the dancer and choreographer Matt Romein, exploring limits of real-time 
avatarization when physical laws are no longer valid (Romein 2016). 

Ifthe metaphor of role and character does apply to the relationship towards the avatar, 
further questions arise: How far can one go beyond anthropological constants? When will 
the identification break? How far can we stretch the laws of physics and biology in order to 
allow some kind of “mental embodiment” or inhabiting the avatar? 

Klevjer refers to a 3-level-model of the avatar by Linderoth (2005). According to this 
model the avatar manifests itself on three independent levels: 

1. A fictive character that you can pretend to be, a role. 

2. A piece of equipment, a tool which extends the player’s agency in the game activity. 

3. A part of the players setting, props which can be used as a part of the players presen- 
tation of self. 


A similar 3-level-model has been suggested for actors (Schwind 1997), but while in the 
avatar, the levels can take distinct phenomenal forms; the actor has only one body that will 


act on all three levels simultaneously. 


CUDDLY TOY, SECURITY 
BLANKET, THUMB 


Finally, the avatar can be characterized as a transitional object in the way Donald Winnicott 
introduced it seventy years ago (Winnicott 1953). This concept is somewhere in the middle 
between prosthesis and model. The transitional object can be part of the own body (like the 
thumb) but the same relationship can be transferred to an outside object (like the cuddly toy 
or the famous security blanket of Linus in the Peanuts Cartoon). ‘The self is projected into 


an object due to a psychological process in the child’s development: 


There is a wide variation to be found in a sequence of events which starts with the 
newborn infant's fist-in-mouth activities, and that leads eventually on to an attach- 
ment to a teddy, a doll or soft toy, or to a hard toy. (Winnicott 1953, 91) 


Winnicott gives a very precise description of this kind of relationship —and he also suggests 


that it is only valid in a certain period of human development: 


1. The infant assumes rights over the object, and we agree to this assumption. Never- 
theless some abrogation of omnipotence is a feature from the start. 

. The object is affectionately cuddled as well as excitedly loved and mutilated. 

. It must never change, unless changed by the infant. 

. It must survive instinctual loving and also hating, and, if it be a feature, pure aggression. 

. Yet it must seem to the infant to give warmth, or to move, or to have texture, or to 
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do something that seems to show it has vitality or reality of its own. 
6. It comes from without our point of view, but not so from the point of view of the 
baby. Neither does it come from within; it is not an [sic] hallucination. 


7. Its fate is to be gradually allowed to be de-cathected, so that in the course of years 
it becomes not so much forgotten as relegated to limbo. 
(Winnicott 1953: 94) 


This also opens up a new aspect of the relationship towards an avatar, highlighting its psy- 
chic function. The avatar in this view serves a specific psychological need; it is a prosthesis of 
the self for a certain time, buffering the heavy emotions that occur in the separation from an 
important person (the mother). This emotional aspect might explain some of the aggression 
that is conveyed through avatars. It can also explain why teenagers identify with avatars so 
much, as well as why avatars somehow “wear out” and get boring when they have fulfilled 


their purpose. 


PETS, COCKERELS AND 
TOTEM ANIMALS 


‘The last metaphor for the relationship towards an avatar puts much more emphasis on the 
autonomous life of the avatar. When conceived as a pet, the avatar has an intense emotional 
connection to the original, it is almost a part of the self, but has its own needs, rights and 
intentions. It may be tamed, but only to a certain extent. Some avatars simulate this kind of 
relationship, like the Tamagotchi-toy that was introduced in 1996, the characters of SIMS, 
that can express recurring needs and intentions, or digital dating simulations like “Date 
Ariane’ (Barnes), that demand attention and courtesy from the player/user/dater. ‘The avatar 
as a half-autonomous being is captured by the metaphor of animals. 

Otto references Clifford Geertz’s description of the Balinese cockfight as a form of 
play, in which the men can take part through their cockerels (Geertz 2005). A very special 
relationship evolves, the men carry their personal bird around, cuddle it, feed it, boast about 
it and talk about it a great deal. The cockerel becomes a half-autonomous external deputy 
of the (male) self. In another context, the metaphor of the avatar as a personal animal also 
might include the idea of a totem animal, which is a personal agent in another world— pop- 
ularized again in the idea of a “patronus” in the fictional world of Harry Potter. 

The metaphor of the pet would indicate some autonomy in the avatar— but not really 
enough to consider it an encounter with the “other”. Does a pet-keeper encounter an outer 
entity when relating to his or her pet? Or is he or she just projecting emotions, needs and 
intentions into a living object? The pet can be thought of as externalized part of the self 
that is kept in some kind of dependency and leading a sheltered life within the wider life 
of the keeper. The metaphor, as with other metaphors above, hints that the avatar is both 
inside and outside of the self. It is this double nature which makes it so hard to grasp. In 
our experiment, we provided metaphors for the participants, discussed them and added new 
ones to get an expert insight into this new relationship. 

Which of the listed metaphors now characterize the new relationship between actors 
and avatars best? Our preliminary results are based on the observation of actors in interac- 


tion with their avatars and our own experiences while acting through an avatar. 


1. OPERATIVE DEPUTIES: 


The outer appearance of the avatar seems less important than the operations we can perform 
with it. No matter which avatar the actors chose and even how similar the avatar was to their 
everyday self, the main relationship developed through performing certain operations. This 
accounts to movements of the head, turning the head, nodding and opening the mouth, 


but even more to performing emotional expressions. Actors were very precise in noting the 
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avatars expressional possibilities and limitations, unsurprisingly, and the buildup of a rela- 
tionship evolved through testing out these possibilities and the restrictions. Indeed, none 
of the used avatars could really satisfy the needs for subtle expression of our test-actors. So, 
while expressive operations seemed to be crucial for the actor-avatar relationship, it was 
also extremely sensitive to disturbances. Even small restrictions in emotional expression led 
to a rather big disappointment and disconnectedness. On the other hand, new emotional 
qualities were discovered through avatars, especially when they had an appearance as ani- 
mals. This seemed to be highly inspirational for the actors and they immediately started 
to play and discover new means of expression. This led to an extension of interaction-time 
between actor and avatar. The simple measurement of time spent in play with a certain 
avatar probably could serve as a valid measure for the quality of the relationship between 
actor and avatar. Is seems to correspond with the emotional operations that can be per- 
formed through the avatar, which supports the hypothesis of the avatar being an operational 
deputy more than a symbolic deputy. 


2. CONTINUOUS FEEDBACK: 


The actor-avatar relationship is based on a continuous stream of operations performed by the 
actor and it ceases to exist as soon as the actor has no sensual feedback of the operations. The 
avatar is not a stable representation of the actor or a role the actor might play, as would be 
a puppet or a mask, instead it relies on permanent feedback-loops. In our setting the actors 
needed a real-time visual feedback of the avatar, provided by a monitor where they could see 
not only the avatar but every operation of the avatar. The relationship, we conclude, is fragile 
and depends on sensual feedback-loops. A third person view of the avatar — similar to look- 
ing into a mirror—seems to be a good setting for this, a first-person view is less effective. 
The identification with the avatar immediately fades, when this feedback is not provided. 
‘This result again supports the operational function of the deputy. 


3. PROSTHESIS: 


The avatar is a way to operate and gain impact, just like a prosthesis and it becomes part 
of an enhanced self of the actor, which makes the metaphor of the prosthesis most apt for 
this relationship. Still the metaphor of the prosthesis is too weak as the avatar is not only 
performing operations that are in the scope of the actor’s bodily self (like a leg or an arm), it 
does not replace a function that already exists, but opens up new possibilities that are beyond 
human expressions. The actor-avatar relationship is most attractive in operations which the 
actor could not perform in their natural human body. Seen in this light the avatar becomes 
an extension of the self like a prosthesis but detached from the physical body while still 
connected to the body self. 

This kind of relationship is new, so words are difficult to find. The term we are look- 
ing for will be somewhere between prosthesis and game character/token/cursor, leaving the 
signified object very free in its appearance but instead describing its operational possibilities. 
Accordingly on a practical and artistic level, processes of operating the avatar and thus pos- 
sessing it or being possessed by it get into the focus. They can create an experience beyond 


biological limitations of our species and beyond the laws of physics. 


POSTSCRIPTUM: 
POSSESSING AN AVATAR. 
OR BEING POSSESSED? 


While the representational function of an avatar seems to indicate that we can possess the 
avatar in the way we possess a car or a dog, the operative—or performative— function 
implies bilateral influences. The operations we perform as an avatar— as operations we do in 
play— can affect the self; they reveal hidden parts of the self, disinhibit unconscious aspects 
and let them take control for a while. It is this revealing aspect of the avatar that makes it 
conceivable that the avatar can lead to new experiences, where the real self does not have 
access to, and thus change the player/user. So, the avatar is not only an instrument to mask 
ourselves but also to show ourselves in a way that we would not dare to “publish” ourselves 
and create experiences that are beyond our usual scope of experiences. In some respects, one 
can say that the avatar takes control of the player/user, possessing them as something that is 
much bigger than the individual’s conscious mind. 

As the operative function relies on the operations of the avatar within its world, story, 


fiction or game, in a computer game these options are highly determined: 


Because the rules need to be implemented by a computer, they need to be expressed 
in terms of an abstract, formal system. The central difference between games and 
computer games is that in the case of computer games, instructions are not instruc- 
tions to the player directly but instructions to the computer. Consequently, the player 
cannot break the rules unless the computer can be manipulated or otherwise made 
to cooperate. This also means that the player cannot relate directly to the instruc- 
tions without somehow sharing the point of view of the computer. (Klevjer 2006: 56) 


Hence, the operative function of an avatar is different from an actor handling a prop because 
it is always predetermined by the omnipotent machine, leading to an identification with the 
machine as much as with the avatar. In the encounter of a player/user/actor with the avatar, 
control seems to be the prime parameter. ‘The first gestures an individual undertakes when 
confronted with an avatar are usually gestures that show that the player has control and 
allow him/her to find out the degree of this control—shaking the head, moving the lips, 
grimacing, etc.—one might call them gestures of possession. Next the original usually builds 
up a relationship by showing emotions— smiling. Or even kissing the avatar— just as they 
would in a real encounter. Many of these gestures could also happen while looking into a 
mirror; in order to build up a relationship with the avatar. 

Otto indicates that the control a player appears to have over an avatar might be 
illusory, because the possession might very well work in both directions. While the avatar 
makes the player feel potent, like a God, “... and makes him believe to control the machine, 
we might also ask, if in reality it is the machine that takes possession of the player.” (Otto 
2013: 115-116). In other words, the player/user will be changed by relating to the avatar; the 
relationship is not one-way. Otto stresses this backlash on the player/user and also suggests 
the idea of possession as a metaphor for the influence of the avatar on the player/user, draw- 


ing on anthropological research: 


In the same way as a mask-dancer the player is also a tool for the avatar and will 
be rewarded by a temporary sublimeness. So if the relationship between player and 
avatar can be understood through ritual contexts like the cock-fight, then maybe the 
practice of this play could be understood as a process of possession. 

(Otto 2013: 182) 
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This leads to acting theories of ritual, liminal experiences and possession, often associated 
with mask theatre. Looking at the avatar as something the player seems to possess, while 
he or she really is possessed by the avatar, is a strange way of thinking, but it accords with 
experiences and theories of acting. The role— even when spelled out in detail as in scripted 
theatre—seems to enable the actor to create experiences beyond their personal scope 
through inhabiting a character in a fictional world. The new relationship to the avatar is 
mediated by the machine/the computer and the discussion above shows that one of the main 


features to be explored is the feature of control: Who is in control and who is possessed? 
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ENGAGING 


CONSCIOUSNESS. 
TIME AND DURATION 
IN HOLLY BYNOE’S 


COMPOUNDS’ 


GABRIELLE A. HEZEKIAH 


Holly Bynoe’s digital collage series Compounds offers an experience of space, time and image 
that exceeds photographic representation. Based largely on archival family photographs 
and found materials, the series is a project that explores questions of displacement, loss, 
rupture and knowing.’ The series seeks to extend ideas of the photographic object through 
an exploration of surface tension. Bynoe describes the work as a kind of “poetic engagement 
with the past” —a recognition that memories are false and family stories complicated—an 
attempt to reconfigure images and render them more complex (Ibid). Her process of digital 
manipulation involves high-resolution scanning and Photoshop to produce a layered image 
of subtraction and abstraction. The result is a peculiar experience of time that I suggest 
stems from duration and spatial organization in the creation of the work. The collage Brian 
is particularly striking in this regard. 

‘The central figure in Brian is a young man in what appears to be a photograph from 
the 1970s. The young man is in swim trunks. There is a house in what would otherwise be a 
background— but it floats and does not appear to be part of the same scene. The writing in 
the top left of the image seems to come from an official text. There are palm trees. All except 
a disembodied brown leg are in shades of black and white — but two sepia “age spots” indi- 
cate the presence of the past. The brown leg is bent but standing and is set against a pinkish 
hue. Behind the leg is a horizontal line of what appears to be a paisley marker—a threshold 
tile of the type found in a domestic space. It partially separates the dense, tree-lined portion 
of the collage from the legs in stasis and stilled motion. In this way, the collage is divided not 
into quadrants but distinct yet overlapping spheres. ‘The spaces between the trees rise up like 
tall buildings. One has a sense of fullness though not of immersion. There is a play on sur- 
face. The scratches seem hand worked. Below the feet— of young man and brown leg—is 
a scratched border. This border extends perpendicularly halfway up the young man’s body. 
His feet face forward, slightly turned out. The elements of the collage are perhaps drawn 
from different locations and different times but they all seem to be happening now. The young 
man faces us in a present moment. It is a peculiar sense of time. This is an image of time. 

I am interested in exploring the particular mode of consciousness that allows this 
image to appear to me as an image of time. I wish to examine that which appears beyond 


the level of representation and of which there may be no physical trace. Holly Bynoe’s 
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Many thanks to Andrea Kunard, 
Cyndie Campbell, Ann Thomas 
and staff at the Canadian Photo- 
graphy Institute and the Library of 
the National Gallery of Canada for 
access to research materials. 


Holly Bynoe, interview with the 
author, 15 November 2016. 
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FIG. 1 
BRIAN (2010) BY HOLLY BYNOE 


DIGITAL COLLAGE, 25 x 35 INCHES. COURTESY OF THE ARTIST. 


Compounds offer a unique opportunity to consider consciousness as—and at— the heart of 
the image. While there are clear connections to the social and political in this work, such an 
approach does not address the encounter with the work’s internal character. This character 
is both more and less than an experience of historical time—and it suggests an attitude 
of openness and engagement that allows for the dissolution of a purely historical time and 
recognition of one more fundamental. This fundamental time of experience is what I have 


set out to investigate. 


THE SOCIAL FACT OF 
REPRESENTATION 


Imperial is a collage that addresses what I would call the construction and inscription of the 
colonial subject. There is the photograph of a woman. ‘The dress and hair suggest a photo- 
graph of the early to mid-20th century. The woman’s face is half light and half dark. The 
passport conveying British citizenship — or subjecthood— is 
inscribed upon it. The light side of the face is covered by signs 
of empire such as the British coat of arms. The dark side is 
covered with the physical description of the passport holder 
and the island of origin— Bequia. This locates the holder 
within a Caribbean space. But this is perhaps an image of 
two women. ‘The hair is precariously positioned—like a wig 
or the hairpiece on a paper doll. The eyes are not quite the 
same and the neck appears to move beyond the established 
contours of the body. This is an exercise in layering and con- 
trast. The background is also divided by shade. The white 
dress and grey sky anchor the image. In the background are 
the sharp angles of metal towers, bridges or scaffolding and 
buildings that seem metropolitan. The original images are 
spliced and arranged to produce a representation of the colo- 
nial British Caribbean subject and her relation to empire. 
While Imperial calls to mind questions of inscription 
and construction, The Fordes points to the constitution of 
colonial subjectivity. The main figure appears to be a sartorial 
representation of colonial respectability and the outline that 
of a colonial gentleman. But he is built up and constituted by 
male and female figures of varying shades, all dressed quite 
formally and facing forward as if for official photographs. 
Bynoe seems to have subtracted the original colour of 
the gentleman’s face, adding disembodied faces and 
figures in its place, and the visage of colonial respect- 
ability is filled in with an undeniable blackness that 
almost exceeds its bounds. The pink hair frames a 
face that is stoic but jagged. There is the sense of an 
emerging timeline—of generations being built into 
a colonial figure—and the exposure of this edifice 
as the underlying black skin beneath Fanon’s white mask. The Fordes presents a corrective 
to history, illuminating a series of relationships that might otherwise go unacknowledged. 
Where The Fordes presents a dense stack of constitutive subjectivity, Pedigree offers 
the deconstruction and unpacking of a portrait. Different in proportion from the others — 


40 x 40 inches square —this collage is clearly intended as a frame. We are drawn to read 


this image as portraiture and also to see the posed subject as 
changing and receding over time. ‘There are at least five photo- 
graphs here. We see layers and expansiveness. Bynoe strips 
the constituent images and begins a process of separation that 
appears as a constellation— portraits floating in a galaxy of par- 
tially subtracted colour and sand-like granules, moving along 
an axis of personal memory. The frame within the frame also 
suggests the possibility of a mirror. The reflection is reminis- 
cent of a series of nesting dolls. They pull away from the viewer. 
How deep and how far back is one prepared to go? In this col- 
lage, Bynoe seems to treat space as an unfixed backdrop for the 
unfolding of personal identity. 

In describing his exhibition Archive Fever: Uses of the 
Document in Contemporary Art, curator Okwui Enwezor writes 
that it “delves into critical transactions predicated on open- 
ing up new pictorial and historiographical experiences against 
the exactitude of the photographic trace” (Enwezor 2007: 11). 
Similarly, we might read Bynoe’s experiments as an interroga- 
tion of received historical narratives through a reworking of the 
photographic trace. Official and unofficial narratives, history 
and memory and the repositioning of the face and figure within 
a complex political and social context all point to thoughtful 
juxtaposition as key to the understanding of the historiographical 
and the self. Images from the archive not only bring 
the personal into the collective and the politi- 


cal— they actively embody them. In these images we FIG. 2 


witness the compounded, multi-layered and jagged /MPERIAL (2010) BY HOLLY 


edges of specific historical processes. These are rep- DIGITAL COLLAGE, 40 x 60 INCHES. COURTESY OF THE ARTIST. 


resentations of social facts, extending outwards from 
the photographic trace 


into the material and 


FIG. 3 
THE FORDES (2010) 
BY HOLLY BYNOE 


DIGITAL COLLAGE, 40 x 60 INCHES. 
COURTESY OF THE ARTIST. 


emotional web of imperial relations. 

But Bynoe’s work also offers a more subtle space for 
interpretation. Beyond specific historical processes in these 
images lies a deeper relationship to the experience of time. In 
Imperial, I feel myself confronted with contemporaneous time. 
The time of empire is reflected in the time of the subject, the 
time of imperial writing and the time of the physical edifice. 

These are simultaneous happenings in 


interconnected locations. In The Fordes, 


I experience a time built up over gener- FIG. 4 


ations, ascending and accumulating in PEDIGREE (2009) 
the figure of the gentleman. It is a time BY HOLLY BYNOE 


DIGITAL COLLAGE, 40 x 40 INCHES. 
COURTESY OF THE ARTIST. 


that lies behind. The events are embed- 
ded. Finally, in Pedigree, 1 am faced 
with a lengthening of time as individual 
images are separated and made to occupy 


an ever-widening space that seems to 


extend to infinity. These descriptions approximate the temporal data that I experience as 


being presented immediately to consciousness. ‘They are not specific to the historical pro- 


cesses addressed in the collages. They suggest a movement within the viewer that engages 


with movements internal to the images themselves. Such are the psychic facts that emerge 


beyond questions of the social and of representation.” 


| 


BYNOE 


See Poivert (2016) for a discussion 
of images as social facts 
(representations) and the image 
as psychic fact of consciousness. 

| am indebted to his analysis. 
https://journals.openedition.org/ 
etudesphotographiques/3594 
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Interview with the author. 


THE PSYCHIC FACT OF 
CONSCIOUSNESS 


| Either you keep to what consciousness presents to you or you have recourse to a 
conventional mode of representation. (Bergson 1950: 66) 


‘The spatial organization within the image in Bynoe’s work corresponds in some way to the 
presentation of time to my consciousness. It teases out an underlying tension that is articulated 
through gaps and absences (Archer 2011). These gaps and absences are rendered visible — and 
communicable— through an experience of chronological time. This is a time that can be found 
in Imperial, The Fordes and Pedigree but is virtually absent in Brian. In Pedigree, the figure 
moves away from the viewer, detaching from the surface of the image along an axis that 
draws the viewer in. It is a timeline. In Brian, the main figure approaches. And the space is 
chaotic. There is no discernible arrangement that points to “before and after”. Space in Brian 
is always filled —with reflections, patterns or scratches — leaving no room for an interpre- 
tation of time in a conventional sense. What I experience in this image is not “time passing” 
(Massey 2016: 53) or a spatialized representation of time but time present and indivisible. 

The spatialized representation of time is Bergson’s manner of accounting for the 
quantifiable aspect of time understood as the chronological in our everyday. He enjoins 
us to attend to the immediate data of consciousness as presented through intuition and 
insight (Pogson 1950: vi) reasserting the primacy of experience and our inner life in the 
face of externalizing abstractions. He makes a distinction between real or concrete duration 
and mathematical or chronological time. Concrete duration is a lived experience of time as 
multiple and heterogeneous. It cannot be measured. Chronological time is a homogeneous 
time of discrete past and present. It is time “so to speak materialized” (Bergson 1950: 127). 
It is made up of boundaries. It is visualized and extends through space. I am suggesting that 
Imperial, The Fordes and Pedigree present us with space that delineates, shapes and quantifies 
our experience of time as contemporaneous, accumulated and extended. These collages are 
legible chronologically. But the figures in Brian are de-spatialized, inviting an experience of 
time as duration happening now, presented as psychic fact or mental image. It is time imme- 
diately presented to consciousness. In Brian, we no longer “project time into space” (Bergson 
1950: 101). Instead, we encounter time as a form of fullness within the image. 

By de-spatializing the original photographs and returning to surface, Bynoe has 
produced an arrangement that now has the freedom to express a fundamental time. The 
removal of conventional space returns us to duration. In this duration we experience time as 
indivisible. Bynoe has brought us to an experience of time in the archive that is not simply a 
representation or reconfiguration of the past. We come to experience what I call an archival 
present—a sense of fullness that is anchored in the intuition of a multiplicity of times hap- 
pening now. This is facilitated by the time inscribed in the making of the image that then 
inheres in the final product. Bynoe describes the experience of time in the production of this 
series— hours of scanning and Photoshop —as reminiscent of an era when subjects sat for 
photographs.‘ In this way, we see digital technology facilitating an experience that we asso- 
ciate with earlier forms of production. I am reminded of Benjamin and the aura— but also 
of Susan Edelstein who writes of “the accretions of time layered into the multiple moments 
of the image” (Edelstein 2007: 11) in the long exposure of pinhole photography. 

‘These are all elements of psychic fact—an intuition of time and experience of dura- 
tion within consciousness. They move beyond representation to engage with the inner work- 
ings of the collage. They suggest an organization of consciousness that touches an organiza- 
tion inherent within the image. But if “duration [durée] denotes not a thing, such as time, 
but the always individual act by which time is retained and prolonged in consciousness” 


(Worms 1999: 96) how does my individual act of consciousness produce an image of time? 


MENTAL SYNTHESIS AND THE 
IMAGING CONSCIOUSNESS 


| But the image is a certain type of consciousness. The image is an act and not a 
thing. The image is consciousness of something. (Sartre 2012: 144) 


Pure duration is “a process of organization or interpretation of conscious states” (Massey 
2016: 53) and Bynoe’s de-spatialization is a form of ordering and organization that brings 
this consciousness to bear. My sense of history— the sepia tones that suggest age and mate- 
riality—is dispersed in the image that Brian presents to my consciousness. It does not 
attach to a figure but floats on the surface of the scene. The text in the upper left suggests 
a memory of institutions — perhaps of education — but it is linked to concrete history only 
by an indirect line that moves across the human figures towards the tone near the bottom 
right. The disembodied leg is behind and beside the figure of Brian. It is not embedded and 
does not emerge. Brian is presented not as concretion but transparency. Through him I see 
palm trees and can barely discern a male figure facing left. Brian coming forward is a present 
stillness. My consciousness of past and present and the transparent link exists in the now. 

Time retained and prolonged is not time extended. It is time synthesized and felt. 
The act by which this synthesis takes place sits just below the level of conscious awareness. 
‘There is an attitude of consciousness that is built up over the course of Compounds. It arises 
in the gaps between layers. It arises through a disjuncture between times — between surface 
and time. Elements of Pedigree drop out of the surface. These subtractions provide an open- 
ing through which consciousness connects. I am encouraged to meet—and to make—the 
image there. In Brian I synthesize prior experiences of divisible time with the time happen- 
ing now. It is incorporated into past and present experience. I do not dismiss the existing 
photographs. They do not disappear. They serve as the ground that reflects this conscious- 
ness to me. Ihe digital breakdown and reconstitution of the photographs allows time to 
emerge as the key subject and object of these collages. In Bynoe’s reworking of surface and 
space, the photograph as photograph—along with the notion of historical time—is dis- 
solved. In its place is a mental image — grounded in physical representation but experienced 
as a reflection on this movement into time. To rework a phrase from Cadava (2001: 38): 
“For a photograph to be read as a mental image, it must encounter a constellation of dangers, 
not the least of which is its own dissolution.” The space-time link — now loosened — opens 
itself up to a relation of consciousness. 

For Sartre (2004: 85), the image is a “relation of the object to a consciousness”. He 
conceived the idea of an imaging consciousness to account for the relationship between an 
external image, such as a photograph or a person present before us, and a mental image 
such as that experienced internally when we grasp a photograph intentionally or imagine an 
absent person in the mind’s eye. Through an imaging consciousness the viewer of a photo- 
graph animates that photograph, producing a mental image. The mental image, therefore, 
is not conjured purely in the mind of the viewer but in contact with and reflection upon 
the photographic image as presented. It is synthesis. We see the photograph but it becomes 
both more and less than that which it represents. As Sartre writes, imaging consciousness 
is “consciousness of an object as imaged and not consciousness of an image”. (Sartre 2004: 86) 
I am suggesting that in this work time itself becomes the object of an imaging consciousness. 
Bergson’s duration considers the ways in which space and time as we habitually understand 
them differ from the experience of time—and Sartre shows how this experience of time 
might be imaged in consciousness. In Brian, in my act of duration, past and present are 
synthesized to become consciousness of a time that is without spatial artifice. This is fun- 
damental time, as imaged. And the imaging attitude provides the mode of consciousness 


through which I produce it. 
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EMOTIONAL 
AMBIVALENCES. 
LOVING AND 
HATING A 
TAMAGOTCHI 


MARTINA HEBLER 


In her book “Alone Together”, Sherry Turkle tells a story about actors and robots that she 
experienced when she was in Japan for the first time in the early 1990s. She reported: 


The problems of elderly loomed large. Unlike in previous generations, children were 
mobile, and women were in the workforce. Aging and infirm parents were unlikely 
to live at home. Visiting them was harder; they were often in different cities from 
their children. In response, some Japanese children were hiring actors to substitute 
for them and visiting aging parents. The actors would visit and play their parts. 
Some of the elderly parents had dementia and might not have known the difference. 
Most fascinating were reports about parents who knew that they were being visited 
by actors. They took the actors’ visit as a sign of respect, enjoyed the company, and 
played the game. (Turkle 2011: 74) 


The Japanese, as Turkle was told, valued the “predictable visits and the well-trained and 
courteous actors.” So, she thought “If you are willing to send in an actor, why not send in 
a robot?” (Ibid.) She does not answer the question directly but makes her skeptical posi- 
tion very clear. Whatever one thinks about actors visiting lonely parents on behalf of their 
children, who do not want to or are unable to do so themselves, it hints at crucial aspects 
that are also significant in the context of human relationships to emotional machines: 

First, both actors and robots, replace someone, in this case a specific person in a 
social relationship; but often social robots replace humans as humans, not as a specific person. 

Second, the feelings of actors and robots are simulated. 

Third, neither the actor nor a robot has a long and close intimate relationship with 
the person they visit. Therefore, they do not have a long history together. Disputes, contra- 
dictions, frictions, and family conflicts will not arise. The visits are friendly and pleasant. 


Everything stays under control, nothing gets out of hand. 
242 


Fourth, these actors bring joy to the elderly. The elderly are willing to accept the actors and 
do not view or bring up the substitution In a negative way. 

From a historical perspective, the fourth point is of particular interest. Robots or 
avatars bring joy to people and people are willing to accept them. This has held true for 
robots for more than a century. Robots were shown at industrial fairs. They were presented 
as spectacles that astonished the audience. These early, very clumsy robots, like e.g. Eric, 
were able to bow and say a few sentences. Even though the range of sentences was very small 
and the audience knew that the robots had been outfitted with a record or a loudspeaker, 
they were enchanted and delighted by them (Bülow, 2007: 57-65). 

Today, in the context of artificial intelligence, we 
are faced with a different category of robots. Nowadays, 
robots seem to be able to individually respond and react to 
their counterparts, which is quite different from their elec- 
tronic-mechanical ancestors which endlessly repeated the 
same phrase to everyone, no matter whom they addressed. 
Thus, human-machine relationships have fundamentally 
changed. 

To describe this change in human-machine relation- 


ships, Sherry Turkle spoke of a “robotic moment”: 


This does not mean that companionate robots 
are common among us; it refers to our state of 
emotional-and | would say—philosophical—read- 
iness. I find people willing to seriously consider 
robots not only as pets but as potential friends, 
confidants, and even romantic partners. (...) We are 
poised to attach to the inanimate without prejudice. 
(Turkle 2011: 9f.) 


Turkle observed a clear difference to the 1980s. In the 
1980s, she argues, “computational objects—robots in- 
cluded—should not be allowed into the realm of human 
relationships”. However, “over the next decade, opinions 
shifted.” (Turkle 2010: 4) 

She calls this a change from a romantic moment to 
a robotic moment. While humans had— in a romantic at- 
titude — previously defended the distinction between man 
and machine and emphasized certain peculiarities of hu- 
mans, such as emotions, they were now ready to 
accept machines as friends. Mark Coeckelbergh 
(2017: 154) suggested that Turkle’s definition of FIG.1 


romance is too narrow: “There is no good reason ERIC, THE FIRST ROBOT (1929) 


to limit ‘romanticism’ to a defense of the human/ 
nonhuman border or the alive/dead boundary.” 
Others criticized Turkle for a conservative atti- 
tude and for sticking to dichotomous thought patterns since she considers friendships with 
machines problematic. She was blamed for not tackling the topic of social robots as “philo- 
sophical opportunities and challenges.” (Gunkel, 2016: 198) 

However, the concept of the robotic moment deserves further discussion, particu- 
larly from a historical perspective. Without doubt, humans’ emotional relationship towards 
machines has changed during the last decades. The Tamagotchi will serve as a historical 
example. And while important research on social robots has reflected on the “other,” (e.g. 
Gunkel/Marcondes/Mersch, 2016) this article will look at the transformation of objects 
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(from passively being handled to actively responding) and ask for the emergence of a new 
category of objects, namely that of responding objects. The decisive question here is what it 


is that makes these objects, such as Tamagotchi, so special, so appealing to humans. 


EMOTIONS TOWARDS 
OBJECTS 


Emotions towards objects are historically anything but new. For several decades, emotions 
towards objects have been the subject of material culture research. Authors such as Tilmann 
Habermas or Mihaly Csikszentmihalyi, to name only two scholars, wrote groundbreaking 
books about the attachment of humans to objects (Habermas 1996, Csikszentmihalyi 1998, 
also Downes/Halloway/Randles 2018). Csikszentmihalyi showed, e.g., how emotions are 
ascribed to things through memories of social situations or persons, beyond their possibly 
low material value and their perhaps questionable aesthetics. 

Tilmann Habermas titled his book Geliebte Objekte (Beloved Objects). As a psy- 
chologist he underlined the central importance of objects for personality development and 
examined their symbolic value and psychological significance. He analyses their functions, 
and identifies self-portrayal, memory, empowerment or influencing of moods, e.g. 

Objects are also fetishes (Böhme 2006), they are sacral objects (Kohl 2003), favorite 
objects, gifts. Often, objects are anthropomorphized. Cars or bicycles get names, they are 
seen as a family member. Car mechanics treat automobiles as organisms that need healing, 
as a recent study made clear (Geuenich 2020, Gericke 2020). 

Piaget has already shown that children of a certain age interpret the properties of 
objects as the will of the objects. He investigated a childlike animism, which serves as an 
explanatory scheme for child development (Piaget 1978: 223). Early childhood animism 
does not distinguish between the living and the non-living. Children ascribe a will, wishes, 
conscious activity or even pain to things or to nature. Piaget cites the example of a little girl 
whose doll has indented eyes one morning, which causes the child great desperation. ‘The 
little girl again and again asks if the doll is hurting (ibid.: 245). 

Piaget also distinguishes between four stages of childhood animism (ibid. 207-240), 
without interpreting these stages as a strict sequence of stages clearly corresponding to an 
age. However, animism starts disappearing at around the age of eleven or twelve. Thus, 
Piaget describes the capability to differentiate between alive and non-alive, between things 
and persons, as the result of a process of childhood development that starts with the child’s 
inability to differentiate and explain animism: “The youngest children are animists without 
being able to consciously explain their attitudes.” (Ibid.: 226) 

Nevertheless, this attribution of a will or an intention as inherent in objects is also 
found in adults. Piaget quotes Théodile Ribot, who observed in 1897: “By virtue of an 
instinctive tendency it is well known, though not yet explained, that man assumes that 
which acts or reacts to him has an intention, a will, a causality analogous to his own; his 
fellow men, animals, and such bodies as imitate life through their movements (clouds, riv- 
ers, etc.).” (Ribot 1897, cited after Piaget 1978: 270) Especially when things do not work 
as intended, adults also tend to attribute a will to them (Cf. Geuenich 2020). Piaget and 
Ribot’s descriptions are of particular interest when the relationship between humans and 
emotional machines is looked at. Adults as well as teenagers, know, unlike children, that the 
interactive devices or social robots are not alive. But they treat them as if they had a will or 
were alive, as will be shown in the example of the Tamagotchi. 

What we see here is the conscious ascription of a will to an object. Of course, emo- 
tional machines do not have a will of their own in the sense of a human will. However, these 


objects behave fundamentally differently towards humans than non-interactive objects do, 


and in doing so emotional machines give the impression of having a will of their own. They 
“respond”, they “answer,” they interact. They develop in dependence on the other, they learn, 
they recognize the other, they seek eye contact. The objects are often designed in such a way 
that they appear lifelike. The toy seal “Paro”, e.g., has wide eyes, a weight that corresponds 
approximately to that of a seal’s, it can make beeping sounds. RealDolls or sex robots have a 
lifelike skin and weigh about as much as a slightly built woman, around 100 pounds. 
‘Therefore, emotional machines constitute a different category of objects. In his study 
on the robo dog “Aibo”, Christoph Scholz suggested to speak of “subject-simulating objects” 
(Scholtz 2015). Here he speaks of objects that are designed in such a way that they are to be 
perceived by humans as subjects, so that a subject-to-subject relationship is experienced. No 
matter if one agrees to speak of subject-simulation machines or not, emotions are experienced 
by interacting with emotional machines, since they behave subject-like. As David Gunkel 
(2016: 213) put it by using the example of the social robot “Jibo”, this “is not just another 
instrument, like our automobile or toothbrush. But he/she/it is also not quite another 
member of the family (...). Jibo inhabits a place in between these two options.” Although 
a car is by no means a simple instrument but quite often also its owner’s pride and joy, it is 
surely necessary to reflect on this new category of objects, as will be argued in the following 


using the example of the Tamagotchi: Objects that simulate subjectivity. Objects that can A 


be groomed by their users. Objects that change and evolve depending on how their users puaa 
treat them. a ae 


ARTIFICIAL COMPANIONS OF 
THE 1990S: TAMAGOTCHI 


In the mid-1990s, the Tamagotchi became extremely popular toys, and not only for teenagers. 
Launched in Japan at the end of 1996 by the Bandai company, 30,000 Tamagotchi were sold 
in the USA within three days in May 1997, and within three months over three million 
had been sold (Allison 2006: 163f.). One year later, the Tamagotchi were available in over 
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80 countries. The small, round plastic egg was brought to life by its users and then had to be 
cared for, looked after and maintained in order to “survive”. It is considered one of the first 
interactive gadgets. Anne Allison called it the “ur-form.” (Ibid.: 164) It was introduced as 
an artificial pet. However, as Anne Allison also emphasized, it was an object between all 
categories. A plastic egg that did not look like a pet, an object that simulated life but was 
artificial, a novel toy but also a “companion” (ibid.: 181). A Tamagotchi required feeding, ; 
entertainment, cleaning, rest periods with the lights turned off, and so on. MG 0: rn 
At the end of the 1990s, children enthusiastically nursed Tamagotchi but adults did 
so, too. Children took it to school, which in turn banned it from their premises. And swiftly, od 


“tamagotchi-sitting” services taking care of the device during school hours were established. 


Children emphasized that they loved their Tamagotchi because they had to take care of it. 

Another decisive factor for children establishing the feeling of having an individual coun- 

terpart was that the Tamagotchi developed different characters depending on the individual 

treatment it received. ‘The plastic egg could become a well-behaved, pleas- 

ant companion but also an uncouth, spitting something. This dependedon FIG. 2 

the personal relationship to its individual user. (Ibid.: 172) TAMAGOTCHI FRIENDS-42805 
Hate and violence occurred in the users’ treatment of the Tamagotchi, ORIGINAL UNICORN. 


as it is with many emotional machines. Of course, violence against things 


is not a new historical phenomenon either. People have pounded on their 
cars or kicked copiers for a long time. Teddy bears have also been mal- 
treated. However, when emotional machines are involved, we see further disinhibition and 


intensification of violence. The robot “Hitchbot”, e.g., was subjected to this. Hitchbot was 
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http://www.virtualpet.com/vp/ 
future/abuse.htm (01.09.2020). 


a hitchhiking robot that communicated with humans and was dependent on the help of 


humans. Many treated him like a friend. Others, however, destroyed him in the most brutal 


way, resulting in talk of vandalism. Reports from “Sexdoll” brothels show that customers 


expressed their worst violent fantasies: dead-looking women were to dangle from the ceil- 


ing, among others. And recently a robot has been developed to facilitate its user’s expression 


of physically aggressive behaviors —it serves the same purpose as the punching bag in the 
past (Moorstedt 2019). 


These examples serve to illustrate the continuity 
with regard to violence against things, which is, 
however, apparently lived out in an even more 
uninhibited manner. Simultaneously, and this 
is very important to note, it seems that violence 
against emotional machines is more difficult to 
be physically expressed. In the context of Tama- 
gotchi, there was a talk of “tamagotchi abuse”. It 
was debated whether and when to add a moral 
component and speak of abuse. Some suggested 
the following behavior was abuse: “Leaving your 
Tam’s light on all night. Discipline it for the 
Wrong Reasons. Not Cleaning them up, etc.”' 
This need for rule-setting in the Tama- 
gotchi user community shows that Tamagotchi 
were abused and tortured. Simultaneously, it 
shows that Tamagotchi were treated like living 
beings, or at least there were community attempts 
to establish moral standards, which followed the 
moral standards for treating humans. Further- 
more, the topic “How to kill your Tamagotchi” 
was fiercely debated during the late 1990s. What 
is interesting here is that people reported that it 
was emotionally difficult for them not to take 
care of their Tamagotchi and even to “kill” it. 
Having a Tamagotchi as a friend meant caring 
for something that seemed to be alive but wasn’t 
alive in a traditional sense. Users fluctuated 
between treating it as a living being or a lifeless 
thing. They had inhibitions about killing it and 
treating it badly. They discussed the category of 
“abuse”. This all points to a hitherto unknown 
category of objects: A category of objects for 
which we have not yet found an adequate form 


of living with. 


FIG. 3 
HITCHBOT BY MICHAEL 


BARKER — HITCHBOT GOES TO THE FAIR, CC BY 2.0 


HTTPS://COMMONS.WIKIMEDIA.ORG/W/INDEX.PHP?CURID=46300428 


MACHINES THAT RESPOND 
AND HUMAN MOMENTS 
AGAIN? 


Machines that seem to respond to their users’ feelings, and which develop individually but 
also differently— depending on their interactions with their human counterparts — repre- 
sent a novel category of objects that have begun to change the human-machine relation- 
ship. Emotional machines are adaptive, thus appearing more human or human-like. ‘The 
1990s therefore marked the beginning of a new human-machine relationship, which has 
become much more obvious today: Siri, Alexa, Paro, Pepper, sexdolls or the hitchhiking 
robot Hitchbot have begun to come every day objects that people interact with as a matter 
of course. As Turkle observed in the 1990s, humans seem to be willing to accept machines 
as their counterparts. However, it is not clear yet what exactly that means. Is a social robot 
a viable option as a companion? As an additional friend? Just an amusing toy which humans 
start to treat as if it was alive? Or do these machines become substitutes for humans, as 
Sherry Turkle claims? We seem to live in a period of transition during which novel objects 
emerge in the world, and during which humans’ attitudes and emotions toward machines 
are undergoing changes. At the same time, we do not know yet how our lives alongside 
emotional machines will look like in the near future. 

However, the current Corona crisis may have made clear how indispensable human 
relationships still are. During the pandemic, a bot was released that was intended to mitigate 
humans’ loneliness. Robots were put in soccer stadiums in order to replace cheering fans. 
Sex robots seemed to be an alternative in times of closed brothels. But none of these robots 
were widely embraced. Therefore, one thing has become most obvious in this ongoing crisis: 
Machines are not welcomed to substitute for human social contact. The longing for visit- 
ing with friends and family, for creating community with peers as partners let the robotic 
moment fade away and made the current time a Auman moment. This also holds true for 


the elderly whose family members were not allowed to visit with them in their retirement 


homes. These seniors did not ask for 
robots, actors or avatars. They were 
waiting for visits by their relatives 
of flesh and blood. They longed 
for human emotions and human 


bodies, hugging them. 
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CONSCIOUSNESS 


PERFORMED 


THOMAS GRUNWALD, MARTIN KURTHEN, 
HENNRIC JOKEIT, ANTON REY 


CONSCIOUSNESS LOST 


| “What ho! my lord! - 
My lord, | say! Othello!” 


When Iago suddenly realizes that Othello has become unresponsive, he knows exactly what 


has happened: 
| “My lord is fall’n into an epilepsy. 
This is his second fit; he had one yesterday.” 


He also knows that it would not be helpful to rub Othello about the temples because: “The 
lethargy must have its quiet course” (Othello, act 4, scene 1). He may be a villain, but Iago 
knows the patient, his conscious and his unconscious states very well. But the stage is not a 
hospital. In real life, physicians and neurologists do not know the patients whose conscious- 
ness they must assess in the emergency room. They do know, however, that patients may 
not be completely unconscious during an epileptic seizure. Instead, they may be alert but 
unable to communicate, or unable to say who or where they are because they cannot remem- 
ber. Things become further complicated when the patient’s consciousness must be evaluated 
in situations in which questions are consecutively posed to the left and right brain hemi- 
spheres. This happens during the “Wada-test,” an examination that is sometimes necessary 
for presurgical evaluations of patients who suffer from medically intractable epilepsies but 
who may be cured by epilepsy surgery. In the “Wada-test,” each hemisphere of the brain is 
anesthetized consecutively to test the language and memory capacities of the contralateral, 
awake hemisphere. If both hemispheres of the brain can communicate, each may answer 
differently. This can also happen during examinations of split-brain patients in whom the 
fiber system that connects both hemispheres—the corpus callosum—was dissected to treat 
otherwise intractable seizures. Such patients may be able to point with their left hand to an 
object that only their right hemisphere can see. When asked which object has been shown 
in the visual field of the right hemisphere, however, the patient’s left hemisphere may just 
confabulate balderdash (see Gazzaniga MS 2000). There are certainly verbal reports that are 
indicative of consciousness, and fortunately, these are the verbal reports we deal with every 
day. It does seem, however, that not all reports can readily be taken as proof of “normal” 
consciousness. With so many caveats, it is necessary to ask whether there is just one kind of 
consciousness that can be impaired or whether there might be different forms of conscious- 


ness. 


CONSCIOUSNESS DECLARED 


Patients who have lost the ability to talk, write, understand, or read are aware of their 
impairment. The same is true for patients who are blind in one eye due to a lesion of the 
optic nerve. By contrast, there are patients who went blind due to lesions to their visual 
centers in the occipital lobes but are not aware of their loss, a neuropsychological condition 
called “anosognosia”. Many lesion studies in patients with localized damage to certain parts 
of their cortices that cause circumscribed neurological or neuropsychological deficits have 
shown indisputably that the human brain is not an all-purpose problem-solving machine. 
Rather, it consists of multiple special-purpose modules—or “organs” as Steven Pinker (1999: 
31) suggests calling them—whose activity may or may not be associated with conscious pro- 
cesses. Based on examinations of patients with perception deficits caused by cortical lesions, 


Michael Gazzaniga argues that: 


phenomenal consciousness, that feeling you have of being conscious of some per- 
ception, is generated by local processes that are uniquely involved with a specific 
activity (...) (Gazzaniga 2011: 65-66). 


Some brain modules mediate conscious perception while others subserve processes to which 
we do not have immediate access. Phenomenal experiences cannot be immediately reported 
although they may be conscious. They are not declarative yet but may become so when lan- 
guage centers—which are normally situated around the Sylvian fissure in the frontal, pari- 
etal and temporal lobes of the left hemisphere—become involved in the activity of the “per- 
ception” modules, thus generating a “unified report” of their processing results. Gazzaniga 
calls this perisylvian language area the “interpreter module.” It is the task and achievement 
of this module to make the contents of consciousness declarative to which it has access. If 
this theory holds true, there should be contents of consciousness to which the “interpreter” 
may not necessarily have access, which would therefore be non-declarative. (‘This theory 
also suggests that non-human and prelinguistic human animals should also have conscious 
perceptions without being able to report them.) 

Why should the “interpreter module” try to give a unified report in the first place? 
The answer appears simply to be because it has been asked to. In fact, the interpreter module 
will always try to give a unified report even if it has no access to any relevant information, 
as in the above example of the split-brain patient (cf Gazzaniga 2000, 2011). From an 
evolutionary perspective, however, such occasional errors are a tolerable price to pay for 
the advantages that a unified account offers to an organism with declarative consciousness. 
Based on the finding that all mental activity is mediated by parallel multitrack processes, 
Daniel Dennett (1993) assumes that the seemingly unified stream of consciousness that we 
experience actually consists of multiple parallel content-fixing events that are not neces- 
sarily contents of our consciousness but altogether represent a stream of multiple drafts of 
narrative fragments. Any such draft can potentially become part of our seemingly unified 
narrative but will only become so when it happens to be chosen in response to probing. Since 
a unified narrative helps to reconcile an organism’s past and prospective future, it pays for 
this organism to probe itself constantly and thus receive a canonical narrative. Naturally, the 
narrative must center on the organism itself, whose “self” is thus (nothing but) the center 
of gravity of the narrative. Although there is normally only one such center of gravity for 
every organism, neurosurgical interventions can create disagreement (Dennett, 1992). If, 
for example, a complete callosotomy—the dissection of the large fiber-bundle connecting 
the two hemispheres of the brain—is performed on a patient with incomplete unilateral 
language dominance, an “alien-hand-syndrome” may occur. In this syndrome, when probed 
it is the “more dominant” hemisphere that provides the (more or less) canonical narrative, 


though it may complain about the involuntary actions of the ipsilateral hand. ‘These actions 


249 


250 


are directed by the contralateral hemisphere, which may have its own—non-declarative— 
center of gravity. That the non-declarative hemisphere can have its own intentions indicates 
that, at least for a while, it can tell its own story, with or without words. Usually these 
controversies can be resolved when both hemispheres learn to communicate via remaining 
interhemispheric connections and the detour of the environment. Eventually, the hemi- 
sphere that is in charge of declaring takes the lead. This may be a good “choice” for the 
organism because verbal narratives are less fleeting, easier to handle and can thus reach 
further into the past and the future. 

A consciousness that can declare itself must have more at its command than con- 
scious perceptions. Antonio Damasio (1999) calls this “extended consciousness,” and its 
scope may span the individual organism’s entire lifetime. Extended consciousness makes 
it possible to relate different experiences to each other and evaluate them in light of the 
organism’s autobiography. To this end, it needs both an autobiographical and an extensive 
working memory. This working memory must have a large enough capacity to hold active 
memory contents for some time that define both the autobiographical self and perceptions, 
pictures or thoughts that reflect external or internal events, it is these two aspects that 
working memory must bring together. Damasio states that “autobiographical selves occur 
only in organisms endowed with a substantial memory capacity and reasoning ability, but 
do not require language,” and he believes that “apes such as bonobo chimpanzees have an 
autobiographical self” (Damasio 1999: 198). Nevertheless, with language humans have a 
tool unprecedented in the history of consciousness that allows them to perform tricks with 
memories and perceptions of which no other animal is capable. 

We have become so accustomed to, and dependent on, language that it is difficult for 
us to imagine a conscious mind without it. Perhaps Helen Keller’s narrative can help us here. 
Helen Keller probably contracted meningitis in 1882 at the age of 19 months, which left her 
deaf and blind. Nevertheless, she earned a Bachelor of Arts degree in 1904 and, in 1908, 


tried to explain how it is to be conscious without language, hearing, and sight: 


Before my teacher came to me, I did not know that I am. | lived in a world that was a 
no-world. | cannot hope to describe adequately that unconscious, yet conscious time 
of nothingness. (...) | had neither will nor intellect. | was carried along to objects 
and acts by a certain blind natural impetus. | had a mind which caused me to feel 
anger, satisfaction, desire. These two facts led those about me to suppose that | 
willed and thought. | can remember all this, not because | knew that it was so, but 
because I have tactual memory. It enables me to remember that I never contracted 
my forehead in the act of thinking. | never viewed anything beforehand or chose it. 
(Keller 1908) 


This is the narrative of a human mind before acquiring language. Because Helen Keller 
could encode lasting entries into her autobiographical memory, there is no doubt that she 
was conscious. By looking back, she can even make us fee/ how it is to act without having 


acquired language, sight, or hearing: 


I remember, also through touch, that I had a power of association. | felt tactual jars 
like the stamp of a foot, the opening of a window or its closing, the slam of a door. 
After repeatedly smelling rain and feeling the discomfort of wetness, | acted like 
those about me: I ran to shut the window. But that was not thought in any sense. It 
was the same kind of association that makes animals take shelter from the rain. (...) 
When I wanted anything I liked, - ice-cream, for instance, of which I was very fond, 
- | had a delicious taste on my tongue (...), and in my hand I felt the turning of the 
freezer. | made the sign, and my mother knew I wanted ice-cream. | “thought” and 
desired in my fingers. If | had made a man, | should certainly have put the brain and 
soul in his finger-tips. (Keller 1908) 


This is a narrative of what Damasio calls extended consciousness, made declarative by sub- 
sequent language acquisition. If we, in a thought experiment, allowed this mind to see and 
hear, if we—to use Helen Keller’s words—put brain and soul also in the eyes and ears, we 
can perhaps begin to sense how it might feel to have an extended consciousness without 
language. If, on the other hand, we imagine this mind without the capacity to create suc- 
ceeding memory entries as foundation of an autobiographic self, we might come close to 
Damasio’s notion of “core consciousness.” This comprises simultaneous representations of 
a perceived (inner or outer) object and of the organism that does the perceiving. Thus, an 
organism with core consciousness perceives an object, simultaneously sensing that it is itse/f 
who perceives and not some other organism. It is almost impossible for us to comprehend 
because we cannot be conscious without language and memory. Or can we? 

Consider golf or tennis pros who play their best game automatically, but start to lose 
when they cannot help wondering about how. Or consider actors or performers who cherish 
their “flow experiences” on stage but then start to wonder about their presence or impact 
on the audience and thus spoil the performance? “One of our enemies is the intellect”, says 


Michael Chekhov about the creative process and the inspiration. 


Inspiration comes when everything is forgotten-the method, the technique, the part, 
the author, the audience, everything. Then the miracle happens. It happens that the 
play, the part, begins to exist independently of ourselves. (Chekhov 1985: 55) 


‘These may seem to be far-fetched examples of experiences that most of us normally do not 
share. But how about driving a familiar route for a while and then suddenly realizing that 
you cannot remember the last minute or two, even if you had to turn left or stop at a traffic 
light? If you have a driver’s license, you have probably experienced something similar. But 


were you really driving unconsciously? Daniel Dennett thinks not. He suggests that: 


You were paying attention to other things, but surely if you had been probed about 
what you had just seen at various moments on the drive, you would have had at 
least some sketchy details to report. (Dennett 1993: 137) 


So it seems that expert athletes, artists, and drivers know exactly what to do, and how to 
do it best, in a non-declarative way. In the heat of the moment these experts may be very 


conscious in a procedural way. 


DECLARATIVE BRAIN 
SYSTEMS 


Language production and perception depend on the perisylvian cortex of the dominant 
(usually left) hemisphere. This area acquires its specific function in the first years of life 
during language acquisition. Before that, the human cortex is still flexible enough to allocate 
other brain regions to language functions if critical areas are damaged. A shift of language 
functions to the contralateral right hemisphere, however, can result not from a lesion of 
cortical “language centers,” but from an early lesion of the left hippocampus, an area inside 
the temporal lobe that supports declarative memory (Weber et al. 2006). 

Numerous studies have confirmed the importance of the hippocampus for declarative 
memory. Memory deficits brought about by damage to this brain structure affect memory 
contents that are accessible to conscious recollection making it possible to give an account of 
them (i.e. they are declarative). These memories include facts and events belonging to one’s 


autobiography (cf Squire and Zola 1996, Tulving 2002), and it seems that hippocampal 
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contributions to declarative memory are so important for language acquisition that the brain 
allocates language functions to brain regions within that hemisphere where the hippocam- 
pal system is intact. This is usually the left hemisphere, but can also be the right if the left 
hippocampus is sclerotic. Conversely, language is such an important tool for declarative 
memory that brain systems mediating language and declarative memory processes can be 


considered as one system supporting declarative consciousness. 


NON-DECLARATIVE BRAIN 
SYSTEMS 


As Endel Tulving highlights, to classify acts of memory systems as declarative or proce- 
dural, it is important to distinguish between thought and behavior. He suggests asking 
oneself whether one can “hold in mind the product of the act of memory” (Tulving 2000: 
728). If the answer is “no”, the act was “procedural”. Thus “procedural memory” concerns all 
forms of non-verbal behavior and is one of the most important parts of the non-declarative 
memory system. Motor learning, as in sports, or learning how to play an instrument do not 
depend on the hippocampus, but on the brain’s motor systems, including the basal ganglia 
and cerebellum. Such skills can scarcely be declared, if at all, which makes teaching motor 
skills so notoriously difficult: Ruminating about a motion sequence can hamper procedural 
memory and has ruined many-a-performance in concert halls and stadia! If thoughts are not 
necessary for these performances, and even interfere with them, artists, athletes, and drivers 
may express their consciousness through behavior. Similarly, when the young Helen Keller 
protected herself from the rain, she did not think about causal or intentional relations but 
was “procedurally conscious”. 

Another kind of non-declarative learning that influences behavior is mediated by 
emotions. Emotions are physiological processes of which we are unaware. ‘They can eventu- 
ally become conscious as feelings but, even if they do not, they can influence the human— 
and non-human—behavior. For example, it has been shown that a patient without a hip- 
pocampal formation on either side, who had been severely amnesic for 15 years following 
herpes simplex encephalitis and who could not learn any new faces or people since then, was 
still able to acquire preferences according to positive, neutral and negative affective valences 
that were attributed experimentally to three people (Tranel and Damasio 1993); the affec- 
tive states that guided him were surely non-declarative. This is true of many physiological 
changes in response to external stimuli, which influence the brain as emotions and can—as 
somatic markers—help us in decision-making (Damasio 1994). Thus, emotions are not only 
non-declarative, but unconscious. Even when they become conscious as feelings, it may 
be difficult, perhaps impossible, to verbalize them so that some feelings are expressed as 
metaphor; ‘ butterflies in the stomach,’ for example. Some epileptic seizures with preserved 
consciousness can elicit a feeling that cannot be verbalized because there is no word for it: 
No language can derive a word for a feeling that cannot be shared because only an individual 
person with epilepsy can experience it from a first-person-perspective. 

One important benefit of emotional learning is that it can help protect us from 
potential dangers. To this end, the brain creates a long-lasting association between fear 
reactions and an object or event that has been experienced as threatening during an earlier 
encounter. This kind of learning happens automatically and unconsciously, but is very effec- 
tive. In fact, this so-called fear conditioning is so effective that too much of it can contribute 
to post-traumatic stress disorders. Conditioning of negative (Le Doux 1996, Phelps et al. 
1998) and positive emotions (Murray 2007) depends on the functional integrity of at least 
one amgygdala, an almond-shaped assembly of neurons situated within the medial parts of 


both temporal lobes just anterior to the head of the hippocampus. The amygdalae in both 


hemispheres belong to the brain’s limbic system, a constellation of evolutionary older brain 
structures that contribute to such diverse, but related, processes as production and percep- 
tion of emotions and feelings, monitoring and control of motor behavior, detecting errors or 
granting rewards etc. One of the powerful methods with which the limbic system controls 
and guides behavior is making emotions accessible to the prefrontal association cortex as 
feelings so that we can talk about them and use them for decision-making. Nevertheless, 
we cannot “decide” to be angry, happy or sad. As Tulving says, we cannot hold in mind the 
products of limbic processes, which are therefore non-declarative. 

Fear-conditioning is not the only way humans learn from emotions; someone able to 
speak a language can also warn and be warned. We usually remember events, people, and 
objects much better when our first encounter with them was emotionally arousing. In all these 
cases, the amygdala takes care of the arousal and thus enhances hippocampal activity, which 
can, in turn, result in lasting memory entries (LaBar et al. 1998). These are the declarative 
products of memory acts because we can hold them in mind as either images or words. Note, 
however, that the declarative memory contents we can recall are events, people and objects, 
not feelings. This dual role of emotions in both declarative and procedural memory may help 


resolve some misunderstandings in discussions of “emotional memory” in the performing arts. 


DECLARATIVE AND 
NON-DECLARATIVE 
COMMUNICATION 


If consciousness is a product of evolution that serves the survival and well-being of organ- 
isms moving freely in their environment, it would be surprising if no signs of conscious- 
ness—or other “kinds of minds” (Dennett 1996)—could be found in non-human animals or 
prelinguistic members of the human species. Nevertheless, it has been suggested that there 
may have been humans capable of verbal behavior but without the declarative, linguistic 
consciousness we know (Jaynes 1976). For now, we won't take this suggestion too literally. 
It will suffice to note that Jaynes discusses how two brain systems—here the two hemi- 
spheres—started to interact during the development of declarative consciousness. Recently, 
Solms and Panksepp (2012) described two other brain systems subserving different aspects 
of consciousness. They argued that, when discussing how human mental functioning is 
embodied, two brain representations of the body must be distinguished. First, there is a 
constellation of (multimodal sensory and motor) somatotopic maps, including the respec- 
tive cortices and modality specific thalamic and cranial nerve structures that represent the 
external body and other external objects that provide conscious exteroceptive experiences. 
Because the perisylvian association cortex is part of this system, we can conclude that it is 
declarative. A second constellation of brain structures, including centers in and around the 
hypothalamus, brain stem, and emotion circuits that overlap with the limbic system, rep- 
resents the internal body. These subcortical structures may generate “phenomenal affective 
feelings of their own” (Solms & Panksepp 2012: 155), and thus another kind of conscious- 


ness, not of objects but the subject of perception: 
We may picture this type of consciousness as the neurodynamic page upon which, 
or from which, exteroceptive experiences are written in higher brain regions. 


(Solms & Panksepp 2012: 156) 


Since animals have emotions that help them guide their behavior (Panksepp 1998), we must 


grant them some type of non-declarative minds mediated by this second, and more centrally — 
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While Michael Chekhov would 
simply state: “In our art we don’t 
have to have reasons. As soon as 
we have to have reasons, we can 
do nothing with them, and then 

it is not art. The actor must be 
able to cry without reason, simply 
because he is an actor. If he can- 
not cry immediately, then he must 
leave the stage. If he has to recall 
the death of his father, poor old 
man, etc., etc., then he is not an 
actor.” (Chekhov 1985: 30) 


localized, constellation of brain structures. This system does not include any cortical centers 
that are relevant for human language and thus is non-declarative, though non-declarative 
does not necessarily mean non-communicative. In fact, social animals have a variety of 
means of communication, including alarm calls that not only signal danger but also whether 
a predator is approaching from earth or air etc. (see e.g. Hauser 1996). These vocal com- 
munications in animals are not mediated by the perisylvian cortical centers used for human 
language, but by brain areas that support emotional processing. For example, the anterior 
cingulate gyrus, an important component of the limbic system within the medial frontal 
lobe, participates decisively in the production of innate emotional vocal patterns. By con- 
trast, the motor cortex and its feedback loops involving the basal ganglia and the cerebellum 
contribute to the production of /earned vocal patterns (Jiirgens, 2009). Data from presurgical 
evaluations of epilepsy patients indicate that both of these systems are still alive and well 
in human brains. While the motor cortex and perisylvian language centers produce speech, 
focal seizures within the amygdala can elicit crying, and seizures within the anterior cingu- 
late gyrus can elicit screaming, swearing and laughing (without feelings of mirth). Although 
the expletives used in this involuntary swearing originate as words from language, ictal 
swearing is not propositional speech, but unconscious, overlearned and automatic motor 
behavior. Nevertheless, expletives result in immediate emotional effects in other humans, as 
do other emotional vocalizations like laughing, sobbing, groaning, crying, and screaming in 
humans and non-humans; effectively these act as emotional remote controls. These vocal- 
izations are, however, not linguistic signs; perhaps not even their precursors. One important 
difference between linguistic signs and emotional vocalizations is their acquisition: While 
words can and must be learned, emotional vocalizations are innate behavior; even children 
born deaf and blind laugh and cry. With words and the appropriate syntax, it is possible 
to form an infinite number of sentences, while the repertoire of emotional vocalizations 
is finite, limited and not generative. Words can help keep in mind the product of working 
memory processes in forms of thoughts so that this product can be an object of thought 
processes again, while emotional vocalizations are behaviors intrinsically linked with the 
associated emotions. For humans and non-humans alike it is difficult—though not impossi- 
ble—to control and suppress these emotional vocal behaviors. Whatever control is possible 
is mediated by a “medial cortical system” of vocalization control (Fitch, 2010), which is 
headed by the anterior cingulate gyrus (Jürgens 2009) and may represent the “voice” of the 
constellation of brain structures representing the internal body (Solms & Panksepp 2012) 
and perhaps the voice of “core consciousness” (Damasio, 1999). 

A new kind of voice control appears first in human primates with a vocal motor cir- 
cuitry that Fitch calls “lateral cortical system” (Fitch, 2010, p 350), which perfectly meets the 
demands of the perisylvian language centers. The lateral cortical system and the perisylvian 
cortex of the dominant hemisphere may together represent the communication center of what 
Solms and Panksepp call the “external body,” and of declarative consciousness. Even though 
we now have defined two independent communications systems of the human brain—a medial 
cortical non-declarative and a lateral cortical declarative system—it is unlikely that the first 
evolved into the second, and therefore it is unlikely that emotional calls evolved into human 
language for two reasons. First, both systems still function fairly independently in modern 
humans. Second, both motor prerequisites for speech and cognitive prerequisites of language 


require language evolution to be free of the innate bondages to specific emotions. 


CONSCIOUSNESS PERFORMED 


When language evolved—whether Homo erectus or Neanderthals possessed a protolan- 
guage, or whether language appeared only later in Homo sapiens—may still be a matter 


of debate. Because non-human primates, however, can use tools to some extent and early 


pre-linguistic hominids made and used tools in a much more sophisticated way, we must 
assume that our last common ancestor—and certainly early hominids—had an extended 
(though not yet declarative) consciousness. Making and using tools cannot be inherited, 
so those who used them must have been able to teach and to learn. Without language, this 
can only have been demonstrated and imitated; an important reason for Donald (1991) 
to call the prelinguistic culture of Homo erectus “mimetic” and emphasize that pedagogy 
must have been important for mimetic cultures. Mimetic pedagogy seems an unproblematic 
method for teaching skills like making and using domestic tools. To teach hunting and 
fighting in this way, however, seems tricky because, without luck, teacher, pupil, or both, 
might not survive the lesson! Sooner or later play acting hunting and fighting will have 
become necessary. In other words, adults had to pretend conscious actions; they had to make 
their pupils imagine an animal to be hunted or an enemy to be fought, and in doing so they 
had to make sure the imagined hunter or enemy did not become too realistic and trigger 


automatic fight-or-flight reactions. Merlin Donald observes: 


Much of the education of children in simple societies is still mimetic in nature. The 
basic vehicles of such training are reciprocal mimetic games and the imitation and 
rehearsal of skills. Children mime adults in every respect, including mannerisms, 
posture, and gesture, they learn the customs and scenarios associated with each 
principal arena of action, and they acquire the manufacturing and survival skills 
essential to the tribal way of life. In addition, children learn a series of subtle limita- 
tions on impulsive behavior in a variety of contexts. (Donald 1991: 176f.) 


If children had to learn to control their impulsive behavior in these teaching sessions, their 
teachers also had to control theirs. This suggests that early hominids could imagine and plan 
future events, and thus had command of a non-linguistic kind of reasoning that permitted 
what Karl Popper claimed for a critical scientific method, namely to let “our hypotheses die 
in our stead” (Popper as cited in Dennett, 1995). The need to control, and not trigger, their 
pupils’ fight-or-flight reactions made it necessary for these early adult hominids to play act 
in such a way that it was possible for their children to recognize the teachers’ p/ay acting for 
what it was; i.e. the performance had to be recognizable as such. Or, as Rhonda Blair puts 
it: “Like a memory, the image is a thing unto itself, just as real as—but not the same—the 
object that triggered it”. (McConachie/Hart 2006: 178) 

We are not the first to suggest such a natural origin of the performing arts. For 


example, David Timson conjectures: 
ple, ) 


Acting, however, is as much a part of human nature as loving, or making war. Every 
day we act out to others the day's events. So perhaps in primitive times, before the 
development of language, theatre fulfilled a need. The need to communicate to the 
rest of the tribe where food could be found, for instance, and re-enact in dumb-show, 
the hunt. (Timson 2000) 


We think, however, that employing play-acting for pre-linguistic teaching purposes may 
have helped emancipate vocalizations from their immediate emotional effects. If actors and 
spectators—teachers and pupils—could reassure each other that they were engaged in play 
instead of a real hunt or fight, it was perhaps safe to “quote” a human—or non-human— 
emotional call. Perhaps play-acting paved the way for some of the first words of humans’ 


protolanguage: 


... if we ask how a species already possessed of imitative skills could come to acquire 
specific word meanings, innate cries could provide fodder for a different category of 
words from onomatopoeia, including words for emotions, for reactions to events, 
and for individuals (e.g. by imitating their laugh). Thus one can reject the idea that 
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innate calls form the seed of voluntary vocalization (...), while accepting that once 
imitation was present, such calls could form models for certain words, although they 
are not words themselves. (Fitch 2000: 392f.) 


Thus play-acting could have contributed to the evolution of both a lexical and—through 
gesture—to a gestural protolanguage (e.g. Tomasello 2008). 

Since declarative memory serves not only to preserve episodic memories but also to 
plan the future, and since language with its recursive syntactical structures provides the 
perfect means to relate thoughts, language evolution may also have accelerated the transition 
from object- and emotion-driven behavior to explicitly (and declaratively) planned actions. 
During the evolution of language as a social system, mnemonic and linguistic competencies 
may not always have developed in parallel, from which some of the Jaynesian discrepancies 
between linguistic behavior and declarative consciousness may have emanated. It is con- 
ceivable that individuals may have been able to verbalize a plan but not yet have been able 
to encode this plan reliably into their declarative memory. For acting, these individuals may 
still have had to rely on object- or emotion-driven motivations, and if declarative memories 
were not yet firm enough to motivate goal-oriented labor in all situations, it might have been 


helpful to rely on external memory markers like totems or pyramids. 


EMOTIONS PERFORMED 


During the transition from emotional calls to linguistic communication, the extended con- 
sciousness of hominids became increasingly declarative. However, while language evolution 
required the emancipation of vocalizations from their immediate emotional consequences, 
perhaps there were situations in which humans missed the direct power that their limbic 
vocalizations exerted over others. In spite of all its advantages, propositional language calls 
for argumentative minds and does not have the unhampered power of emotional remote 
control. That this process of emotional detachment has disadvantages may have become 
evident during teaching by play-acting. Then, as now, there probably was a pedagogic “arms 
race” between the teachers’ need for their pupils’ attention and the pupils’ inclination to lose 
interest. For those who play-acted—the first representatives of the performing arts—it was 
perhaps sometimes necessary to again rely on emotions and intersperse the teaching act with 
occasional “real” alarm calls. The need for emotional arousal, or perhaps Jaynesian attempts 
to sidestep declarative argumentations and to tap directly into emotional inspirations for 
actions, may also have motivated the performance of rituals. 

The necessity to elicit emotions was seen by rhetoricians over 2000 years ago. Their 
instruction manuals even classified emotions, the purposes for and the means by which they 
could be elicited according to the hierarchical system of the “decorum”, in which not only 
gestures were designated higher or lower rankings between the poles of the “sublime” and 
the “low”. The most sublime and important gesture and figure of speech, the aposiopesis, 
was to break off a sentence leaving it unfinished, accompanied by a spreading of both arms, 
which even threatened to let the toga slip down (Mühlmann 1996). The orator showed him- 
self to be completely exposed, so overwhelmed by his feelings that he was unable to further 
verbalize his thoughts. This sublime gesture and figure of speech came not without peril 
for the orator; he ran the risk of making a fool of himself if either his emotional appeal to 
the audience was inappropriate to the subject of his speech, or if it did not work. The orator 
had to make sure that he succeeded in remote controlling his audience’s emotions and, to 
this end, the rhetorical instruction manual taught that the orator must really experience the 
emotions he wanted to elicit in his audience. Verbal communication, with all its connota- 
tions and prosody, was not enough for this sublime purpose, it had to end and let the body 


talk (by means of the medial cortical, limbic system). 


This rhetorical technique brings several methods of acting to mind, training to reach the 
unconscious by conscious means. Based on the teachings of Konstantin Stanislawski, fur- 
ther developed by Lee Strasberg, Michael Cekhov to name but three, a method was created 
that enables actors to recall emotions from their own biographies to identify with their roles, 
whether on stage or in a movie scene. Judging from neuropsychological memory theories, 
this must seem to be a contradiction. Recall is a method that retrieves an item stored in 
declarative memory, i.e. a ‘product of the act of memory” (Tulving 2000: 728). However, as 
we have shown, emotions are unconscious, and feelings non-declarative memory processes 
that cannot be recalled. But we also showed that conditioning can pair selected triggers with 
emotional responses. To find out whether top-class actresses and actors can really elicit their 
own feelings on demand, we performed a study using functional magnetic resonance imag- 
ing (fMRI), a technique used to show active areas within the brain. We asked participants 
to imagine ten different scenes of their repertoire (for 30 seconds each) with all the empathy 
and emotion they would use on stage. Then we compared their brain activity elicited by this 
task with the actors’ brain activity when they rehearsed the text of the same scenes without 
being emotionally involved. We found that 6 out of 9 actresses and actors participating in 
this study activated the amygdalae in their temporal lobes and a ventromedial part of the 
frontal lobe that is also part of the limbic system. So, it seems that experts within the per- 
forming arts may be able to pair their feelings with characters or events on stage or on a set. 
(Whether “the method” is the only way to do this, or whether intensive examination of both 
oneself and the character to be impersonated can accomplish the same thing, is not within 
the scope of this chapter.) 

When the Institute for the Performing Arts and Film and the Swiss Epilepsy Centre 
joined forces to study whether actresses and actors can recall not only emotional episodes, 
but the actual emotions, we had no idea that this study would take us on an emotional jour- 
ney halfway around the world. 

Many have experienced being deeply touched by actors and actresses while watching 
a play, but how can they convey such complex feelings? And do they really “love” or “hate” 
their partners while acting, or are they merely pretending? 

Arts and sciences have a long tradition of quarreling over such questions like ‘hot’ 
or ‘cold’ acting, emotional identification or distant simulation. Using modern techniques of 
imaging brain activity, a joint research program Authenticity of Emotion,—from which the 
lecture-performance Act Like You Mean It evolved—has examined whether actors use emo- 
tions in their performances. Aided by Shakespeare’s Romeo and Juliet balcony scene, we 
presented our findings as part scientific lecture, part performance. But how could we per- 
form research results, especially when they come from such a complex subject as emotions? 
Emotions, however, are precisely what performers claim expertise in. 

To summarize the tour de horizon: it was the local context and intimacy that allowed 
the spectators to identify with the actors and characters. What we initially considered a dis- 
advantage—working with local performers for the home audience in their local language— 
became a real advantage; it drew the spectators into the dilemma between scenic attraction 
and scientific explanation. One conflict in the back and forward between theory and art 
resolved itself only at the end, when the play could be seen and felt as a piece in a closed 
course, when head and stomach would, momentarily, correspond perfectly. 

Nevertheless, for those of us who are not successful, professional performers, it is not 
possible to recall emotions. We can recall emotionally relevant events, and thus most of us 
can understand what ‘Trevor Nunn meant when he wrote in the lyrics of “Memory”, a song 
from the musical Cats: “I remember the time I knew what happiness was”. But recalling the 
associated emotion, i.e. “re-fee/ing” happiness (or sadness, anger, pain, or love) is difficult 
for most of us. This is probably one reason why early and modern declarative cultures have 
always cherished the arts. If not to artists, to whom else should we turn when we want to 
have feelings elicited deliberately? After all, they have been using emotional remote controls 


since the time of mimetic cultures. In contrast to rituals and rhetoric, however, the perform- 
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ing arts have had to use their emotional remote controls together with a declarative safety 
net, which protects us from immediate fight-or-flight reactions. Some declarative safety net 
would still appear to be indispensable, even for the modern performing arts. Without this, 
the audience might mistakenly perceive the villain on stage as authentic and flee, or even kill 
the actor, which allegedly happened to an actor who played Iago too convincingly (Epstein 
1993). Rituals and rhetoric, in contrast, do not play-act and have no use for emotional safety 
nets. They deliberately use whatever emotional remote controls they can access. 

The audience, then, wants the performing arts to create emotionally moving events. 
We should not forget, however, that, like novels, dramas are not least narratives in which 
plans of actions and conceptions of lives are dramatized alongside emotions. To convey the 
“full mental monty,” the performing arts must address both our declarative and non-declar- 
ative consciousness. If our selves are the centers of gravity of our own narratives of con- 
sciousness, then other narratives show us other centers of gravity or, as Iris Murdoch put 
it: “The purpose of literature is to show that other people exist.” If drama catches us both 
declaratively and emotionally, it might even let us witness alternative conceptions of our 


lives, which we may let die on stage in our stead. 
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ARTIFICIAL 


INTELLIGENCE. 
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EXPECTATIONS 
AND A NEW 


CATEGORY OF 


MACHINES’ 


MARTINA HEBLER 


In his short book, “Mein Algorithmus und ich” (My Algorithm and I), the writer Daniel 
Kehlmann writes about his experiences with an algorithm named CTRL (Kehlmann 2021). 
He had received an invitation from a Cloud Computing Company to write a story together 
with the algorithm. Kehlmann enthusiastically accepted the invitation since he expected 
to get a glimpse into the future: If he—as a writer— was to become obsolete in the near 
future, he would be the first to know. Now, what did Kehlmann learn about the future? 
Although the algorithm produced some impressive sentences Kehlmann concluded 
that it was not possible to jointly write a story with this algorithm. CTRL was unable 
to conceive of a plot, and it started to produce nonsense after a relatively short time. As 


(4 


Kehlmann writes: “... then the wheels jam, the transmission is blocked, then it’s over, and 
you have to start a new story.” (Ibid.: 48) 

Two things are remarkable here. First, Kehlmann uses a classical machine metaphor 
to describe the failure of his experiment with AI: Wheels jam, the transmission is blocked. 
Second, during his narration he repeatedly comes back to one of his initial expectations, 
i.e., that CTRL could become a writing companion, a counterpart alongside whom human 
authors could create a story, the way they would be able alongside a human writer. How- 
ever, Kehlmann’s unexpressed prerequisite was that he expected to encounter a humanized 


technology as his counterpart. 


BR 


A short version of this article 

was first published as follows: 
Künstliche Intelligenz: Eine 
neuartige Kategorie von Maschine 
oder die Vermenschlichung der 
Maschine / Artificial Intelligence: 
A New Category of Machine or the 
Humanisation of the Machine, in: 
Keskintepe, Woschech 2021. 
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Recently, “the machine” or “tech- 
nology” have been the subject 

of different monographs and 
articles. Cf. e.g. Burkhard 2018, 
who examines the void of a 
philosophical theory of machines, 
Schatzberg (2018), Technology. 
Critical History of a Concept, 
Chicago/London; Popplow 

(1998) who follows the history of 
machines in early modern time; 
or for a short overview cf.: Heßler 
2020. 


3 


On the metaphor of the mechani- 
cal clock cf. Mayr 1986. 


4 


The thesis of three different 
categories of machines can only 
be sketched briefly here. A longer 
work is in preparation. 


5 


The distinction between the 
trivial machine and the non-trivial 
machine is not that simple, as 

von Foerster argues. However, it 
cannot be elaborated on that in 
the realm of this article. See also: 
Kaminski 2014. 


This leads to the core question of this article, namely to what extent AI represents a new 
category of machines and if so, what kind of machine? The focus here is not only on the 
transformation of concepts of machine. Inseparably intertwined with that is the change in 


human-machine relationships and the change in human self-image. 


HISTORICAL CONCEPTS 
OF MACHINES 


The concept of the machine is currently experiencing a remarkable renaissance.” Within the 
context of digitisation and Artificial Intelligence, there is even talk of a “second machine 
age” (Brynjolfsson/McAfee 2014). But what is astonishing is how matter-of-course this 
has been. Why do we talk of AI as a machine at all? After all, AI does jar with traditional 
notions of machines. It represents a completely new category of machine. But what kind of 
“machine” is AI in the first place? In fact, is it even a “machine”? 

In the early modern era, the mechanical machine became a metaphor for processes 
that were regular, regulated, and reproducible (“running like clockwork”).? Furthermore, 
in the 19th century, the concept stood for smooth-running, rational processes. As Joseph 
Corn (2011: 29) stated: “Clocks spread the technical ideal of machines that run themselves 
with minimal control or intervention by a human operator.” These notions shaped the image 
of the machine right through to the second half of the 20th century as something that was 
regular, uniform, always the same, reliable, dependable, and predictable. The mechanical 
machine performs according to comprehensible and clear rules. 

And even if, until recently, computers and AI continued to correspond to this 
image of the programmed, rule-based machine (Heintz 1993), a new dimension to the 
machine concept had already been identified as early on as the 1950s, precisely at a time 
when computers were “evolving”. In the 1960s, the philosopher Gotthard Günther (1963) 
spoke of the “transclassical machine”, which represented a new category of machines. Such 
a machine no longer performed work, as the “classical Archimedean” machine did, but 
processed information (ibid.: 183). Similarly, Max Bense (1955: 7) emphasized that the 
“mathematical machines (...), occasionally also called thinking machines,” represented a 
“new state of being of technology” (ibid.: 8). Regarding the “thinking machine”, French 
cyberneticist Louis Couffignal (1955: 13) stated that machines were “of the most diverse 
kind”, which is why machines had to be classified using different categories. 

‘Thus, the advent of the computer in the 1950s made it necessary to identify and 
categorize these new types of machines.* Now, the talk turned to a new “state of being” 
of the machine. It put the spotlight on computers’ ability to process information instead 
of them performing work. Further, it referred to the difference between mechanical and 
mathematical machines. 

However, Heinz von Foerster (1993: 244-252), also inspired by cybernetics, added 
another thought beyond the rule-governed nature of machines. He developed the concept 
of a “non-trivial machine”, which he distinguished from “trivial machines”. The trivial 
machine resembles the mechanical machine by working in a rule-governed way, being com- 
prehensible, and predictable. The non-trivial machine is a machine whose behaviour is not 
predictable and not understandable.° Thus, Heinz von Foerster anticipated—regardless of 
the various different technological foundations— characteristics which distinguish today’s 
AI from other types of machines, as will be further explained in a moment. His concept of 
a non-trivial machine can therefore not be subsumed under the two categories of machines 


already mentioned, since it points ahead to present AI conceptually. 


THE NOVELTY OF Al 


Given the latest trends in Al and its pervasiveness in everyday life, the history of the machine 
concept needs to be updated again. A third category of machines has emerged: Machine 
learning represents a categorically different and novel form of machine. 

Perhaps the current renaissance of the machine concept thrives on the hype that 
surrounds machine learning, which is the mainstay of present-day AI successes. The catego- 
rial novelty is featured in the name itself: adaptive and learning machines are what we are 
now dealing with.’ They learn by using data; the algorithms are self-improving. Program- 
mers have ceased programming how an algorithm ought to arrive at a solution; instead, 
they program algorithms capable of learning from their “experience” and then developing 
a model by themselves to apply to other data. They are, therefore, developing autonomously 
(Kersting, Lampert, Rothkopf 2019: 19). 

From a historical perspective, this is not something entirely new. Neural net- 
works were already being discussed in the 1940s and 1950s. Further, cybernetic machines 
were also “learning”. Research into machine learning has been conducted since the 1980s 
(See overview Lenzen 2018). The fact that AI is now celebrating new successes is down to a 
new type of data volume, high processing speeds, and a new quality of efficient algorithms. 
It means that AI is no longer comparable to 20th century machines. But what exactly is so 
novel about it— beyond its essential “learning” aspect? The following will give a description 
of seven characteristics which are mutually dependent. 

First, AI interacts in a new way with its users. Let’s return to the opening example 
of writing a story jointly with AI, and this becomes very clear now. The mediating role 
of technology in the writing process has already been addressed many times. People have 
written with fountain pens, typewriters, or computers, which has influenced and altered the 
writing process in each case (See overview: Gaderer 2020). However, writing a story jointly 
with an algorithm, represents a completely new dimension— even if AT’ possibilities are 
currently still limited, as Kehlmann aptly put it. Many other examples for human-algo- 
rithms cooperation could be added. AI has become an advisor, a partner, and an assistant 
in everyday life. AI can no longer be seen as a mere tool used to edit, create, or calculate 
something. Decisions are now made, discussed, or written in conjunction with AI, as part 
of a densely interwoven network of actions. 

Here, and this is a second central key aspect, AI applications develop individually 
while interacting with its counterpart. AI is adaptive. Siri, Alexa, chatbots, social robots, 
and my previous example, the algorithm CTRL, learn the behavioural patterns, matters of 
interest and preferences of their counterparts and, over time, “advise” and “respond” more 
and more precisely, in keeping with those individual habits. Historically, this is a remarkable 
turning point. The machine-like aspect is no longer that which is standardised and ever 
constant. Instead, AI systems evolve differently. They have their own individual “bio- 
graphy”, one that depends on their counterpart and the context in which they are used. 
It is the everyday behaviour, the everyday use by each individual that permanently modifies 
the AI. 

Third, this in turn implies yet another characteristic that differentiates AI from 
machines in the traditional sense. The evolution of algorithms that learn autonomously 
cannot be predicted or planned, a feature of AI that resembles von Foerster’s concept of a 
non-trivial machine. How AI will respond and interact with human beings in a year’s time 
is anyone’s guess. This means that machines also have a future all of their own. 

Fourth, and connected with that, their results are not reproducible, another funda- 
mentally new characteristic compared with the sort of machines we have been familiar with 
so far, which had no surprises in store; indeed, we could rely on their “sameness”. However, 
talk to a bot today, for instance, and you quickly realise that the same question produces 


different answers. 
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Kaminski and Gelhard (2014) 
have published an edited volume 
in which the authors also put 
forward the thesis of a new cate- 
gory of machine. They coined the 
term “informal technicization” 
(informelle Technisierung) and 
focused on a new human-machine 
relationship in which technology 
becomes imperceptible because 
it cannot be directly experienced. 
The focus lies on ubiquitous com- 
puting. 


Cybernetic machines were also 
learning and adaptive machines, 
but different ones. Cf. researches 
such as Cordeschi 2002; Pickering 
2010; Müggenburg, 2018. 


261 


262 


8 


https://gi.de/informatiklexikon/ 
explainable-ai-ex-ai (02.02.2022) 


Fifth, another new dimension concerns machine decision-making. Admittedly, this is not 
something new. Decision-making machines have a long history. As early as the 1950s, 
philosopher and writer Günther Anders mocked the use of a computer to decide whether 
the US should end the Korean War. The computer, according to Anders, ultimately made 
a more ethical decision than humans did (Anders 1988: 59-64). Norbert Wiener (1958: 
174-180) warned against “government machines” already in the 1950s. In the 1970s, the 
Allende government in Chile launched a cybernetic experiment in an effort to control the 
Chilean economy (Medina 2011). However, what is new is that AI, when it is applied in 
everyday life, must make ad hoc decisions. AI does not only support humans in decision- 
making processes. In part, it must make decisions itself— within seconds. This is what is 
currently being debated, e.g., when it comes to autonomous driving or the use of drones for 
military purposes. This raises the question of “moral machines.” (Misselhorn 2018) 

Sixth, and closely connected, what’s also new is that machines are now capable of 
developing biases. Depending on the database used in each case, AI applications can, as is 
currently debated, perpetuate discrimination. Research into the history of technology has 
often shown that technology— by virtue of its cultural nature—is never neutral and that 
human assumptions have always been incorporated into its development. However, the fact 
that machines develop a bias in the course of their “actions” and then reach culturally formed 
decisions that are not transparent has a lasting impact on the very image of machines. In the 
1950s and 1960s, it was hoped that computers would be able to make rational and objec- 
tive decisions in an irrational world (Erickson 2013). ‘Today, AI applications favour certain 
groups of people while discriminating against others. 

Seventh, and finally, artificial neural nets are something of a black box for AI devel- 
opers. The fact that technology is a black box is by no means historically new. During the 
20th century, technology increasingly became a black box since most of its users no longer 
understand how it works. For example, while the first automobiles required their operators 
to possess a certain level of technical expertise, cars increasingly became a complex black box 
that even trained mechanics could only understand and repair with the help of diagnostic 
tools. Nowadays, most everyday technology is a black box for most users, be it the washing 
machine, the computer, or the mobile phone: a black box that can be easily operated or used, 
but whose functionality is not understood in everyday practice and, most importantly, also 
does not need to be known or understood. What is new, however, is that AI has also become 
a black box for the AI developers themselves. As Klaus Mainzer (2019: 254) put it, “From 
an engineering standpoint, authors therefore speak of a ‘dark secret’ at the heart of machine 
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learning AI.” He cites a 2017 article: “... even the engineers who designed (the machine 
learning based system) may struggle to isolate the reason for any single action.” (Ibid.) 

Since the early aughts of the millennium, the term “explainable AI” has emerged, 
and with it, a debate about the possible consequences of its incomprehensibility. In the 
meantime, developing explainable AI constitutes an important goal for computer scientists 
and societies. Within computer science as a discipline, however, this problem has been a 
topic of discussions for a long time. The term “black-box models” refers to this phenomenon, 
which goes hand in hand with probabilistic modelling and statistical learning methods, 
such as deep learning.* Thus, machines have a secret, as it were, in that their behaviour is 
not always “comprehensible”. What’s new is the incomprehensibility in principle that distin- 
guishes Al as a black box from the many black boxes of consumer technology that we use so 
“naturally” without really understanding them. 

Altogether, this gives rise to a breathtaking image of AI that radically breaks with 
previous notions of the machine: i.e., a machine that interacts with people, that “responds”, 
that is adaptive, and develops in keeping with the counterpart in each case. Further, it is 
capable of developing bias. It is meant to make moral and ethical decisions; its behaviour 
changes; and it is not always reproducible and not always comprehensible, even for its devel- 
opers. It becomes our advisor, companion and/or assistant. It is the interplay of all these 


characteristics mutually conditioning one another and rooted in machine learning that 


makes current AI systems an entirely novel category of machine. In a nutshell, one could 
say that Al is becoming human-like. AI no longer corresponds to the 20th century image 
of the machine which humans often chose to distance themselves from. Rather, AI appears 
humanlike in that its behaviour is not predictable, that it is not necessarily reproducible, in 
that it may develop biases, and in that it has a past, so to speak: its behaviour is based on 


past data. 


PARADOXICAL EXPECTATIONS: 
TRADITIONAL NOTIONS 

AND A NEW CATEGORY OF 
MACHINE. 


During the 20th century, Western discourses on technology concerns focused on the fear 
that humans might become machine-like. Up until the 1970s, critical voices uttered dread 
that machines would lead to totalitarian rationalization, de-subjectification, and that uni- 
formity of humans would ensue. People though insisted that they were different from 
machines. Behaving like a machine had clear negative connotations. 

As argued above, AI no longer corresponds to this machine image of regular, 
mechanical, and always “perfect” machines. AI now simulates many features that have 
earlier been emphasized as typically human in contrast to the mechanical: individuality, 
subjectivity, emotionality, the unpredictability of human actions, the human capacity for 
ethical action. That means that AI challenges the human self-image, insofar as humans still 
define themselves in contrast to the machine. 

Further, AI as an apparently “humanized” category of machine challenges human 
expectations towards machines. Humans tended to expect “high standards” from machines, 
e.g., a machine should be devoid of bad human characteristics, it should do “better” than 
humans. Early on, the pioneering AI researcher John McCarthy stated that nobody wanted 
a computer that loses its nerve (Cited in Lenzen 2018: 244). Or, to put it favourably, it 
should be a companion, just as Daniel Kehlmann imagined. 

However, human expectations for AI as an “objective” advisor or a reliable companion 
point to a paradox: Al’s described properties, which make it a new category of machine and 
apparently human, simultaneously mean that AI loses many of the properties of the classical 
machines, which humans have come to appreciate: the expectation of humans exerting 
dominance and control over machines, the assumptions of objectivity, reliability, predict- 
ability, and rationality. Exactly these qualities were attributed to the classical machine. 
However, these same properties cannot be ascribed to AI as a novel category of a machine. 

Concepts of traditional human-machine relationships and traditional images of 
machines still shape expectations towards AI. That is particularly true for an anthropocen- 
tric attitude that sees humans as controllers, describes technology as assistance, emphasizes 
the hierarchy between humans and machines, and thinks in terms of dualisms. In a nut- 
shell: Historically determined concepts of humans and machines shape our expectations of 
Al—-although they are no longer suitable. 

It seems that humans have not fully grasped yet the ambivalence and novelty of AI. 
They expect a categorically novel machine to behave in the same manner as the classical 
machine did. Daniel Kehlmann fell into the same trap when he commented on the failure 
of AI to write a story jointly with a human; his measuring stick was the concept of a clas- 
sical machine against which AI failed miserably. It seems that people are not aware of this 
paradox: On the one hand, Al is supposed to become more human-like, and its new proper- 


ties such as adaptability, learning ability, development ability, and individualization indeed 
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make it apparently more human-like. On the other, however, AI is supposed to be reliable, 
transparent, predictable, and objective. However, if AI becomes more human-like, it will 


also display more of what is regarded as human “failure” and unreliability. 
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THE EYES 


HAVE IT. THE 


PROBLEM 


THE AVATAR 


GAZE* 


OF 


ALEXANDER GERNER 


1. NOTES ON THE FACIALITY 
MACHINE: ILLUSION OF 
VERGENCE OF GAZE AS THE 
ILLUSION OF LIFE? 


The synthetic gaze that I call the “avatar gaze” is never boring. How could it be? It is not a 
true gaze at all but a synthetic media-simulated gaze. There is no gaze if you are rendered 
sleepless 24/7 (Crary 2014). There is no gaze if there is no rhythm of life and death, sleep 
and wakefulness, insomnia and awakening, interest and boredom, focus and distraction, 
selection and election, but an uncanny existence between life and death. There is no gaze 
if attention is not allowed to fall asleep and be woken up. ‘There is no eternal gaze. A gaze 
comes to life, gets tired, gets bored, and ends. 

A significant Disney study is related to robotics and the creation of immersive tech- 
nology in which the simulated synthetic gaze plays an important role: the realistic inter- 
active robot is artificially and mechanically programmed as if having a gaze. Hereby, the 
introduction of simulated saccades is meant for an illusion of vergence —the ability to focus 
on objects both near and far—to simulate the gaze at you instead of non-animated “empty” 
eyes looking “through you” rather than “at you” (Pan et al. 2020). 

This also implies research on how to synthetically create the illusion of being looked 
at by the human counterpart through automatized animatronic a) breathing movements, 


b) blinking, c) gaze following, and d) ocular movement. All these micro-movements have to be 


1 ‘This research was financed 

by Portuguese national funds 
through FCT — Fundagäo para 

a Ciência e a Tecnologia, l.P., 
within the scope of the Transi- 
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Hacking Humans. Dramaturgies and 
Technologies of Becoming Other. 
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FIG. 1 


SCREENSHOT FROM DISNEYRESEARCHHUB (OCTOBER 19, 2020). 
“REALISTIC AND INTERACTIVE ROBOT GAZE” 


YOUTUBE VIDEO RETRIEVED FROM: HTTPS://WWW.YOUTUBE.COM/WATCH?V=D8_VMWWRJGE 


FIG. 2-5 
SCREENSHOTS FROM DISNEYRESEARCHHUB (MAY 27, 2019). 
“PRACTICAL PERSON SPECIFIC EYE RIGGING” 


RETRIEVED ONLINE: HT TPS://WWW.YOUTUBE.COM/WATCH?V=SQOGOTUC8R5G. 


Do automata ever get bored or fall asleep? 


mechanically reproduced, e.g., by quick motor-redirections, based on mathematical models 
and their technical implementation for creating the “illusion of life” through animism (Gygi 
2018) and animation (Eisenstein 2010). Eisenstein understood the fascination and longing 


for animism as an antithesis to “soulless geometry and metaphysics” (Eisenstein 2010: 118). 


Disney’s research focuses on the analysis of automatic spectator boredom (Deng et 
al. 2017) and animatronic mimicking of gaze. Disney Research is also tackling the problem 
of gaze in mathematized digital animations. As each person has different characteristics of 


the eye—eyeball shape, interocular distance between the two eyes, rotation center (distinct 


from the geometric center of the eye- 
ball), and specific visual axis (different 
from the optical axis)—this anima- 
tion research concentrates on the eye 
motion characteristics of each individual 
that is reproduced as a believable digi- 
tal character. The approach includes eye 
tracking and gaze estimation, capturing 
and modeling the eyes, and eye rigging 
estimation (Bérard et al. 2019), using 
50 eye poses that reconstruct the overlap- 
ping averaged eye poses of limbus, eyelid 
interface, and visual axis—resulting in 
animation-ready eye motion characteris- 


tics of each animated face. 


Visual axis defines the gaze and is not identical with the optical axis. To neglect this differ- 
ence results in cross-eyed gazes and uncanny valley effects due to inaccurate eye movement 
animations. Eye rigging for simulating eye motion characteristics for each Avatar includes 
taking into account the fact that the eye is not perfectly spherical and therefore does not 
exhibit rotation around all axes. This means for the accuracy of the eye estimation that a 
person-specific eyeball shape, rotation center, interocular distance, visual axis, and other rig 


parameters resulting in an animation-ready eye rig have to be considered: 


“a) The eye is controlled by six muscles (two per degree of freedom), which operate 
in a complex orchestrated way to rotate the eye. b) The gaze direction is not aligned 
with the optical axis of the eye (...) but corresponds to the visual axis (...), which is 
formed by the ray passing through the center of the pupil originating from the fovea 
at the back of the eye, which is the area where the retina has the highest sensitivity.” 
(Bérard et al. 2019: 442) 


2. WHAT IS IT THAT IS 
MISSING IN THE COMPUTER- 
ANIMATED FACIALITY MASK 
AND SYNTHETIC GAZE? 
CAN WE PUT IT BACK? 


| “Let it be assumed that lines drawn directly from the eye pass through a space of 
great extent (...)” Euclid, The Optics of Euclid (1945: 357) 


“The radiating gaze /Euclid’s ta optikd (300 BC) can be read as the ethical comple- 
ment to his much better-known geometry. The book deals with rays emitted by the 
eye. These rays are something for which both words and sense have been lost.” 
Ivan Illich (1995: 52) 


The gaze is a phenomenon that awakens and falls asleep; it is based on the surprising and 
spontaneous election (Gerner 2011) of what grabs our attention in the world and cannot be 
reduced to a programmed, pre-selected set of possible entities. In the gaze, a visible differ- 
ence exists in the comparative synchronization of the same and the other. 

As the avatar moves forward and acts as ifa human with faciality, it must be criti- 
cally scrutinized as it transposes our imaginative openness of gaze towards the other into 
prosthetic concrete movement— and that the gaze of the one meets the gaze of the other. 
This encounter of regarding in a vis-a-vis means, for Plessner, reciprocity between me and 
the other in which we can switch to the position of the other and become an other for each 


other: 


(...) the gaze that meets the gaze of the other. In it we have the elementary phe- 
nomenon of reciprocity between me and the other. As soon as my gaze meets the 
other’s eye, | see myself looked at, gazed at — and not only (in the manner of the 
ophthalmologist) the eye is examined. The other not only has a look, but gazes at me 
and thus stands in the position of the vis-à-vis, as the one with whom I can exchange 
positions. In this interchangeability of the point of view, which his gaze testifies to 
me, he is an Other, I am an Other for him. (Plessner 2016: 384; translation mine) 
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2.1 THIS GAZE DOES NOT EXIST: ON THE MISSING 
GRAVITAS OF ATTENTION IN THE AVATAR GAZE 


Ethical, aesthetic, and social difficulties derive from normative issues in computer-assisted 


and modeled ghosting of already deceased famous actors in cinema, such as ghost acting 


(Winick 2018). In this case, face tracking, face matching, and face transfer use computer 


FIG. 6 


DE-AGED CGI SELF-AVATAR FACE OF THE 
ACTOR ROBERT DE NIRO IN THE IRISHMAN. 


SCHAEFER, S. (SEPTEMBER 2, 2019) “THE IRISHMAN: DE NIRO 
RECREATED GOODFELLAS SCENE TO TEST DE-AGING CGI,” 
SCREENRANT. HTTPS:/SCREENRANT.COM/IRISHMAN-ROBERT- 


DE-NIRO-GOODFELLAS-DE-AGING-TEST/ 


generation to display the face of a dead actor 
through a mask as a digital over-coding onto the 
face of another actor for the digital reenactment of 
expression, mouth, gaze, pose, or even the whole 
body. The techniques of deep neural networks 
using deep fake technology create believable 
media (Mirsky & Lee 2021) that are applied in 
digitally altered film acting. Another example is 
de-aging. In this technique, an actor works with a 
self-mask wearing his or her younger digital face- 
mask (where the problem of young face-older body 
dissonance is easily noted as uncanny—as was the 
case for the character played by the actor Robert 
DeNiro in Scorcese’s The Irishman (2019). Scorcese, according 
to the Guardian newspaper, commented that there was a prob- 
lem with the expressiveness of his actors’ eyes, adding: “Does 
[the technique] change the eyes at all? If that’s the case, what 
was in the eyes that I liked? Was it intensity? Was it gravitas? 
Was it threat? And then how do we get it back? I don’t know.” 


(Scorsese, cit. in Salmon 2019) 


As in the case of paintings where an animal’s eyes seem to follow us, gazing back on us and 


stimulating our imagination, we experience the fact that the animal gaze has gravitas. In that 


case, we believe that the animal other’s gaze is responsive, weighed by temporality and mor- 


SITTING OPPOSITE OVER 1500 MUSEUM VISITORS, 
OF WHOM PORTRAITS WERE TAKEN: “THE ARTIST IS 
PRESENT” moma, NEW YORK, 2010. 
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tality, and possibly sparking an interest between estrangement 
and trusted familiarity, gravitating around awareness, curios- 
ity, seriousness, uncontrollable and unexpected danger, that 
may surprise us. The sense of face, if the gaze is well captured, 
provides a sense of rendering the possible other’s presence 
real and visible—inside an anthropological-evolutionary 
advantage of enhanced human visibility (as a hunter), but as 
well with an increased risk of counter-violence in becoming 
visible to the gaze of the other, 
e.g., as prey. 
FIG.7 A gaze responds dif- 


ABRAMOVICH’S  ferently in each new scene. 
PERFORMANCE en In the que of 
OF THE GAZ E, ne other, oe eee 
CONSISTING tial, directional, and textural 
OF NEARLY data as noise becomes social 
750 HOURS OF co-resonance with the other. 


It shows our face to the other 
and holds the other in our 
regard, but “with the oth- 
ers eyes” (Plessner 2017). A 


stranger can reciprocate a gaze 


that becomes an exchange: currency of the social and cultural necessities of per- 
ception open up a spontaneous field of modes of possible encounters — sociality is 
possible. The modality and cultural-historic regimes of gaze prompt responding 
modes of action. Thus, the gaze endows us with the other’s attention, forcing us to 
act accordingly. In so doing, we hold each other responsible in one another’s gaze. 
A gaze without gravitas would be emptied from the possibility of pairing 
the gaze with the other where almost nothing happens— but this “almost nothing” 
is the gaze that happens face-to-face with the other, hooked on the necessary 


imagination and mimesis, as the other looks at you. An example of an art of pure 


gaze would be the Abramovich performance: 
“The artist is present” that is strictly about pure 
gaze and the openness of what it possibly could FIG. 8 


trigger as Dre Te: 7). THIS GAZE DOES NOT EXIST: SYNTHETIC 
A solipsistic, sleepless, endless parallel IMAGE WITH AVATAR GAZE CREATED BY 


avatar gaze that fails to create a face-to-face 


situation, a shameless or riskless gaze: this is a 


“GANISM” — A GENERATIVE ADVERSARIAL 


synthetic faciality mask that has no face to lose NETWORK STYLE GAN2 SMILING FACIALITY 


(see fig. 8). A mathematized surface of synthetic RECONSTRUCTION cr. WANG 2019; KARRAS ET AL. 2019 


faciality looks through you—“withdrawn” RETRIEVED FROM HTTPS://THISPERSONDOESNOTEXIST.COM, 
into endless infinity. The avatar gaze would SEPTEMBER 6, 2021, CREATED AT THE TIME OF 15:54 P.M. 


be an extinguished gaze from and towards 
nowhere—one that does not /ook at you but 
redirects you to the same: the avatar holds a 


gaze from nowhere. 


As there are no dreams that the avatar gaze dreams up, we want to look more closely 


at Scorsese’s statement and the complexity of what he designates as a faulty deal: he wanted 


a younger actor’s face and got a CGI face with an attentional engine that does 
not show the temporal experience of its actor, but rather an animated and 


mathematically constructed faciality mask that exists outside of time. 


2.2 THE EYES HAVE IT? 


Even animated faces and their eyes become issues in the dispute over their 
acceptability and their uncanniness. A case was made by fans against the 
designed (and behavioral) human-like hyperrealism of eyes and the imitation 
game of human-likeness in CGI with the cinematic adaptation of the Sega 
videogame Sonic in the film Sonic— The Hedgehog (2019). In two different 


versions of test trailers, fans were upset about the appear- 
ance — too close to human eyes— of the character and in 
a subsequently accepted criticism the director Jeff Fowler 
(@faultown) announced on Twitter on June 2, 2019 in a 
message that received 175,000 hits: “Thank you for the 
support. And the criticism. The message is loud and clear... 
you aren’t happy with the design & you want changes. It’s 
going to happen. Everyone at Paramount and Sega are 
fully committed to making this character the BEST he 
can be...” #sonicmovie #gottafixfast. 

If Robert de Niro lends his face to a de-aging CGI 
treatment, we might want to call his younger digital face 
a faciality avatar. Algorithmic rationalities (cf. Mersch 


2021), including mathematical modeling in the culture 


FIG.9 a> 
SCREENSHOT OF THE MOVIE 
SONIC — THE HEDGEHOG SHOWS TWO 
VERSIONS OF THE TRAILER — ABOVE 
WITH UNCANNY EYES BEFORE THE 
CRITICISM AND BELOW AFTER THEY 
WERE CHANGED. see: KINOCHECK.COM (2019) 


HTTPS://WWW.YOUTUBE.COM/WATCH?V=FM2SV YAQYFU 


of AI, expand machine learning and artificial neural networks that work with statistics 


and algorithms into (yet) non-mathematical realms, creating among others programmed 
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sociality (Gerner 2020; Bucher 2018), while at the same time aiming at artificial creativity. 


Algorithmic rationality in digital faciality, despite producing “similar” faces and deep fake 


facialities of persons who do not exist, is limited by what cannot be decided by a program. 


FIG. 10 
FIXING THE UNCANNY EYES 

OF THE CGI CHARACTER IN 

THE FILM ALINA: BATTLE 
ANGEL BEFORE (FIG. 10, LEFT) 
(TWENTY CENTURY STUDIO 
SINGAPOR 2017, FIRST TRAILER) 
AND AFTER (FIG. 10, RIGHT) IN 
THE FINAL MOVIE VERSION 
2018 WITH ENLARGED PUPILS 
AND EYEBALLS THAT ARE CON- 
STRUCTED TOPOLOGICALLY AS 
EYE-SPHERES. DO LARGER EYES 
RENDER COMPUTER-GENER- 
ATED EYES MORE “REALISTIC”, 
MORE ACCEPTABLE AND LESS 
UNCANNY IN THEIR DIFFER- 
ENCE TO HUMAN EYES? 


In this sense the website “This person does 
not exist” wants to alert us to the existence 
of non-human human-like images created 
by AI algorithms, but are these algorithmic 
facialities able of a human gaze? Might we 
confuse an animated image with a gaze of 
another living creature? 

Such an algorithmic image could 
be called an image singularity, as it merely 
comes into existence when the site is clicked 
upon; in the next click the algorithmic data is reorganized to become yet 
another different automated image-output—to create another “person” that 
does not exist. 

‘The attribution of non-existence lies in the eyes of the spectator who, by 
his or her technological involvement in the visualization algorithms, actually 
co-triggers the image that does not refer to any real-life person whose gaze 
one could meet. However, the biggest problem of social interaction in shar- 
ing attention — sometimes called “the eyes have it” (Emory 2000) — remains 
unresolved by AI. Facebook is currently attempting to tackle this problem 
with the high-fidelity Codec Avatars that simulate social presence, considering 
faces and using coordinated audio- and gaze-driven animations for photo-re- 
alistic real-time relighting of animatable face avatars (Sai et al. 2021). 
Facebook’s integrated facial, eye, and speech AI algorithm remains in the 
realm of the simulation and animation of facial landmarks and facial maps. 
Physiologically, the white sclera prepares humans (Kobayashi/Koshima 2001) 
for social attention, known as “joint attention” in social psychology, anthro- 
pology and philosophy. Still, the mutual gaze is different from the physio- 
logical reality of the eyes as part of a sense of face. The social “other” in this 
situation of joint attention—a key feature of human social cognition—in a 
mutual gaze encounter triggers a categorically different social function than 
the mere physiological biology of visibility of an object provided by the visual 


organ of the eyes. Joint attention implies cognitive, behavioral, and phenom- 


enological aspects including common, mutual and shared attention (Siposova & Carpenter 


2019), involving knowledge, goals, and shared affectivity. Can these sensible and fragile 


situations of joint attention be simulated by an AI model that includes an avatar with a 


simulated gaze? 


If we delve into the question that the filmmaker Martin Scorsese is asking —what 


precisely it is that we relate to if we look at the eyes of the other, such as the intensity or even 


“gravitas” of an actor’s gaze—we should look at the interesting empirical research on the 


issue of the extramissionist attribution of attention to the other as a fundamental problem of 


the social in social perception of gaze and visual joint attention. 


2.3 CAN AVATARS SIMULATE THE EXTRAMISSIONIST 
ATTENTION BELIEF THAT THE EYES PROJECT 
LOOKS OR EMIT RAYS OF LIGHT? 


Can we think of a folk psychological model we use permanently in a social co-attention 


and joint attention situation in which the “eyes have it” (Emery 2000)? This hinges on a 


socially strong condition of attention capture and attention maintenance with the proximity 


of the other. A possible, but unorthodox explanation of “the eyes have it” could be an until 
now unknown fundamental physical force embedded in the eye-gaze— in parallel with the 
cosmological physics debate on the possible existence of an until now unexplained dark 
matter of quintessential force that would theoretically push the universe faster apart than 
contemporary physical explanations. Could it be possible that there exists a special force by 
which our eyes, through their gaze, emit light on objects in space? 

Recently, Arvid Guterstam et al. (2019) showed that despite explicit knowledge 
of the test-persons that the eyes neither emit light nor any kind of force-carrying beam 
in an extramissionist sense of attention, we are nevertheless subconsciously prone to the 
claims of these metaphysical-metaphoric models. Humans believe even subconsciously in 
the force-emitting capacity of the other’s gaze. We endow the other’s gaze with a power 
over objects in a commonly perceived world. This nonsensical notion of an attentional 
emitting gaze able to affect objects at a distance nevertheless has to be read within “the 
view that the sense of agency relies on causal inferences between actions and effects” 
(Recht/Grynszpan 2019: 7). Although this specific extramissionist attention beam does 
not exist, its imaginary-experiential strength reflects how we construct counterfactual— on 
the physical and physiological level— metaphorical everyday experiential and intentional 
folk beliefs regarding the social power of proximity and our perception of the gaze of the 
other, resting not upon a foundation of a third-person physical fact, but on metaphorical 
experiential-psychological first-person evidence (in our case on the social importance of the 
gaze and shared attention for our social behavior). As Edward B. Titchener already noted at 
the end of the 19th century, such beliefs are “based on a misinterpretation of fact” (Titchener 
1889: 895) in which we become susceptible to the attribution of the other actor’s attention with the 
interpretation and belief of a force-imposing amplification of the other’s gaze. This means 
that the constitutive look of the other that constitutes me as a person, the foundational act of 
becoming an intentional object in the gaze of the other, is related to the sensory imagination 
of a metaphoric beam or spotlight based on the mere reality of visibility through light. 

This imagination of the emission of light through the eyes is a false popular belief that 
may create in the observer an “uncanny feeling” inscribed in a cultural history of invisibility 
still to be written, as the idea of extramissionist eyes emitting light is contrary to the func- 
tion of the eyes as part of seeing, linked to the visual nerve and the different brain circuits 
involved primarily in receiving and not emitting light to form the images seen. 

If we think in terms of the experiential field approach of Kurt Lewin, we have to 
heed the transformation of a peaceful landscape into a battlefield (Lewin 1917: 447) and vice 
versa. This phenomenological switch indicates how peaceful zones may quickly turn into 
combat zones of social-psychological danger that suddenly turn from an idle or aesthetic 
appreciation of open landscape into a hunter’s gaze. 

What is crucial in Kurt Lewin’s trailblazing description of the field psychology of 
the war landscape, published in 1917, is the clear-cut distinction between two different 
kinds of landscapes: on the one hand, the mathematical-topological and vector-concept of space 
that is open— principally in all directions/vectors and equally — io infinity, and now, on the 
other hand, the “directed landscape” (gerichtete Landschaft) that replaces it. This “directed 
landscape” is an intentionally and experientially structured landscape or lived and limited 
space that has to be heeded as a socially experiential lived place and contains not light beams, 
but calculated vectors of the other’s active and open attentional gaze, which we never catch 
(Finkielkraut 1997: 12-13). Catia Pesquita and colleagues found that “humans are sensitive 
to subtle differences in bodily cues that occur when someone else’s attention is controlled by 
an internal choice vs. an external signal” (Pesquita et al. 2016: 1). 

‘The activity and empathetic relation to mathematized actions I call externally 
“programmed action.” Programmed action is a forced transposition of intentionality: a 
“programmed other.” This programmed other— paradoxically— suspends the characteris- 
tics of alterity. We might observe such a characteristic alterity suspension in the application 


of DeNiro’s de-aged mathematized facial mask in The Irishman. Scorsese calls it a loss of 
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WHAT IS 


KOINONIA UNDER 
TECHNOLOGICAL 


CONDITIONS? 


DIETER MERSCH 


Over the past fifty years, the core of the sociality of the social has culminated in discus- 
sions within the social sciences in the reconstruction of what Aristotle (1999: 1252a) called 
xoLvwvia, or more exactly xowwvia nomtxh. Koinonia politikë determined the con- 
cept of politics in Antiquity. However, it should be remembered that nodeg, the polis as 
an organized polity, served first and foremost the formation of community, so that the 
political and the social came together. Yet the literal meaning of the Greek word xowovia 
is the association, xowvdc the public, or xotwwvéw to share or to participate so that the 
expression addresses what is common to the community, what holds it together, as Plato 
(1976: 449a-466d) wrote in Politeia. In particular it refers to political organization, which in 
Aristotle’s concept comes before social bonding by creating the prerequisite legal order.! 
TIoAtc and xowovia were thus synonymous for Aristotle and his successors, for in Anti- 
quity it went without saying that bonds between human beings could not be permanent 
without law. Somewhat later, during the rise of Christianity, the concept underwent a reli- 
gious turn, so that xowwwvia referred to “participation” in the kingdom of God, in God’s 
creation or in the divine state, creating the true re/igio, the bond between the divine and the 
human, or between transcendence and immanence. For that reason, xoıvwvia plays a key 
role in the New Testament and in the dogma of the Church fathers. The law in this case was 
not always made by humans but was part of the presentation of the divine revelation of the 
tablets to Moses. 

These delineations have all been lost in modern sociology and social sciences. 
xotvwvta, chronically questionable in modern societies, must therefore first be newly 
defined, whether within a classical social contract or by means of power and violence, 
which however has failed to prove its legitimacy. It is striking that all social designs since 
the catastrophe of the Second World War and the totalitarianisms of the twentieth cen- 
tury, have explicitly revisited the question of xoLvwvia in order to give it new foundations. 
Among the most noted attempts to respond to this question in political philosophy are, 
aside from communitarianism, Jean-Luc Nancy’s concept of “being-in-common,” Niklas 
Luhman’s “social systems,” Jürgen Habermas’ theory of “communicative action” (Verstandi- 
gungsverhältnisse) and Bruno Latour’s Action Network Theory. The latter three in particular 
shall be discussed in the following as exemplary examinations of the problem of xoLvwvia. 


under specific technological conditions. 


1 Arendt (1958: 32) would however 
claim the opposite, that “man is 

a ‘social’ before he is a ‘political 
animal”. 
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A good overview of the debates of 
the time is collected in the cata- 
log edited by Diederichsen Franke 
(2013). 


3 


4 


This was also the birthplace of 
the conservative communitarian 
movement that brought attention 
to the blind spot of liberal 
societies, namely a lack of solidar- 
ity and neighborliness that cannot 
be guaraneed through individual 
freedom. 


See on this in particular the 
Habermas/Luhmann debate in 
their book Theorie der Gesellschaft 
oder Sozialtechnologie (1971), 

esp. Habermas’s contribution on 
p. 142-290. 


Luhmann (1973: Chap 2, 4) interprets xowvwvia systematically, that is, as embedded in 
the functions of social differentiation and their stabilization. With this there is a shift in 
the canonical concept of xoLvwvia, which is redefined in an analysis of the mechanisms 
that make differentiation possible. Sociality and community are not simply givens but 
must be created or produced through functions that ensure the reproduction and preser- 
vation of constitutive systems of difference in the social. To these belong in particular trust 
and communication, as well as media as a generally formative power, whereby it is not up 
to the individual functions whether community is formed, but up to the functioning of 
the systems themselves (Luhmann 2021, 2017). Against this, Jean-Luc Nancy picked up 
Martin Heidegger’s figure of “being-with” (1962: § 25-27) and wove it into his ideas about 
“being-in-common” with the emphasis both on the shared experience and participation as 
well as on the “with,” that is to say allowing for both on the cum and the com of communio. 
xoLvwvia is thus grounded in a possibility of participative cooperation, which Nancy under- 
stands as deseuvrement, an “unworking” or “inoperability” that cannot be created but is 
always still to come (Nancy 2010, 1990, 2000, 2016). Yet déseuvré means less “unworking” 
than “not-working” in the meaning of interrupting a “work” or renouncing production, or 
also turning away from every kind of working and towards that final practicality of society 
and community. In that case, for Nancy, deseuvrement refers to the element of becoming 
within xowovia, which simultaneously retains its own difference or distantiality (Abstan- 
digkeit). Put another way, the possible commonality of human beings is always yet to come. 
It is unfinished and torn asunder by a fundamental rift (Riss), so that it is not the work of 
a working process or production, but must first be formed with all of the connotations of 
forms and forming, also including the refusal of the form or its unrealizability. 

It remains a constant question what then participation might mean, how it can be 
produced and whether it is enough for an adequate xowwvia. Participation in the most 
radical sense of everyone taking part was for example one of the fundamental tenets of alter- 
native movements in California in the early 1970s. In their determination to avoid any usur- 
pation by “the system,” they organized themselves in new “communes” and other forms of 
living together.” What that might look like again brought up the question of xowwvia, which 
they however sought to solve within the framework of a social cybernetics that did not use 
political or social or religious arguments, but solely technological ones. It was above all the 
utopia of computerization as a universal tool of connection that should realize the reorgani- 
zation of sociality through the virtual networking of a global public information system.’ 

In a remarkable essay still topical today, Jiirgen Habermas, at almost the same time, 
rejected this technical solution and castigated social cybernetics as a protofascist form of 
“social technology” that sought to control everything.’ As an alternative, he propagated the 
linguistic turn that reinterpreted xowwvia within the practice of communication. Haber- 
mas’s philosophy of understanding (Verständigung)— a “theory of communicative action” 
that can in fact be read as a “critique of communicative reason” — is not a design for a utopia, 
but a concept of communicative Enlightenment on the basis of a social discourse in which 
everyone participates (Habermas 1984, 1990). Habermas (1979) underlines two aspects of 
this model: first, the theory of the dual propositional and performative structure of speech, 
which sees language as having both a representative as well as a relational or binding func- 
tion. Through this “illocutionary force,” sociality is constituted as a simultaneity of mean- 
ing and practice through presuppositions such as comprehensibility, truth, truthfulness, 
and rules. Second, the process of coming to an understanding is for Habermas intrinsically 
linked to an ideal but contrafactual rationalization. This holds the “unforced force of the 
better argument” against the violence and terror of forced obedience (Habermas 2001: 94). 

Habermas propagates xoıvwvia in the sense of a permanent process of enlighten- 
ment generated by dialogue that brings members of a society together through reason. In 
this way the sociality of society follows the “regulative idea” of a process of understanding 
that is both continuous and consensual. This model undoubtably contains its own form of 


one-sidedness. One the one hand, it is questionable whether xoıvwvia is rationally expli- 


cable at all and must not instead always be thought of as xowæœvia notah which has the 
political and not the rational as its starting point. Furthermore, in Habermas’ model, its 
binding force is limited to the pragmatics of language and communication. But not only 
are these subject to multiple biases, the model all but ignores the many other social prac- 
tices such as art, gift giving, rituals, or also witness, law, and trust, not to mention the 
libidinous human connections which should not be underestimated in the creation of bonds. 
Furthermore, the sole alternative between reason and violence as construed by Habermas 
does not exist, because there are situations in which consensual discourse cannot help but 
itself communicate violence. Clearly, xowavica must be understood as a dilemma, for law 
can not be enforced by consensus alone just as conversely there can be no legitimacy without 
social consensus. 

Nevertheless, Habermas offers a decided counter-program to the counter-culture 
computer network program model, which aims to regulate the public sphere solely through 
unlimited access and the xoıvwvia by providing an open infrastructure, as if the commons 
alone would make community possible. Since the 2000s at the latest, this former utopia has 
flipped to reveal the opposite of these hopes: a monopolistic power structure of social media 
that are helpless in the face of the proliferation of hate speech, disinformation, and the 
destabilization of democratic structures.’ Only the naiveté of techno-euphoric subcultures 
and their stubborn visions of transparency, freedom, and equality could believe that fast 
internet connections, good will, and wide-scale participation would be enough to allow the 
quasi natural development of a globally connected community of knowledge counter to the 
hegemonic political orders. Technical networks are in fact diagrammatic structures ruled by 
formal relations not suited to the growth of xowovia, so that the realization of “happiness” 
through technology alone remains a fatal misconception. 

Bruno Latour (1993) introduced a new meaning for the general metaphor of net- 
works, which he placed under the aegis of a “symmetrical anthropology” that understands 
sociality not as “humanist” but as an intrinsic connection between “human” and “non- 
human entities.” In his book Reassembling the Social, he also refers back to the xowwvia of 
Antiquity, but expands formal participation to include the heterogeneous engagement of 
both humans and other creatures or things, artefacts, signs, goods, technological devices 
and media, etc. (Latour 2005: 5ff.). In Latour, “networks” have nothing to do with computer 
networks but act as an abstract concept of shared agency. Despite all varying readings and 
vulgarizations of actor-network-theory, the term “actor-network” is therefore one word, in 
which actors are not those who act or interact with one another, but a description of diverse 
kinds of agency that stem from different modes of relational connections, multiplicities 
of very different elements. Here that which Latour previously, in his analyses of scientific 
research, referred to as “nets” of disparate hybrids, become “associations” that are as loose 
as they are open, mixed coalitions that do not allow discrimination among the elements or 
types within them (ibid.). 

Decisive here is the expansion of the idea of the social to include the simultaneous 
participation of the non-human and the non-social, for, as Latour wrote together with 
Michel Callon, “There is no thinkable social life without the participation—in all the 
meanings of the word—of nonhumans, and especially machines and artifacts.” (Callon, 
Latour 1992: 359) Similarly, he wrote in Reassembling the Social: “We are no longer sure 
about what ‘we’ means.” (Latour 2005: 6) Instead, we are dealing with heterological pro- 
cesses of bonding, intertwining, and connecting, but also of separating and a paradoxical 
entanglement made of links that do not in themselves even need to be social. Hence Latour 
localizes xoıvwvia in a radical relationalism in which, as Erich Schüttpelz has aptly shown, 
relations are primordial in contrast to the relata to the extent that the former first gives the 
latter their weight and their position. It thus follows that xowovia can coexist with very 
diverse “modes of existence,” for which reason Latour proposes replacing the in his view 
problematic term “social” with the term “collective.” (Latour 2013) Collectives can mean 


a variety of things without immediately privileging human “associations” or social classes. 


5 In a remarkable move, Facebook 
founder Mark Zuckerberg, defend- 
ing free speech in front of the 
American Senate, insisted that he 
only provided an infrastructure. 
This is extraordinarily naive, for 
infrastructures are never neutral, 
but themselves play a role in 
shaping the type of expression 
possible. 
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On the concept of a Verständi- 
gungsgemeinschaft or Kommunika- 
tionsgemeinschaft see Apel 1986. 


The reason for this critical maneuver is obvious: on the one hand, Latour is attempting 
to dodge the seeming foundational paradoxes of sociology and the social sciences —that 
sociality in the sense of xowovia must always be presupposed to determine the social. 
It thus proves in advance to be an idealization of what is to come the reality of which is 
chronically questionable. This is true of the concept of “cooperation” as well as of “social 
relationships” or “communication,” all of which presuppose that which they are seeking to 
explain, for example when the concept of communication assumes a delineation of a prede- 
termined “community of understanding” of speakers and listeners (Verständigungsgemein- 
schaft) and their mutual recognition through the practice of speaking.‘ Just as conjuring the 
term “sociality” gives no reason for the fact of its existence, the term “communication” does 
not tell us why we speak to one another or what moves the speaker/listener to listen one to 
the other. This is incidentally also the crux of theories of the social contract, to the extent 
that every contract demands rules of negotiation and decision that in turn must be based on 
that which the contract is supposed to regulate. We are thus dealing with a petitio principii 
that however puts logic before xowwwvia, because it does not recognize the specific tran- 
scendentality of the social and thus the precedence of xotvwvia. Rather, people are always 
dependent on sociality, without which they would not be “human.” Therefore the foundation 
of the social cannot be discovered, at its bottom is always another social. The social cannot 
be avoided, but the more important question is whether the concept of “networks” is suffi- 
cient to describe it. 

Rather than founding a “sociology,” Latour only formulates an “ecology of collec- 
tives” about which it remains unclear whether it is already a social science, of which the 


glossary states under the term “Association”: 


Extends and modifies the meanings of the words “social” and “society,” words that 
are always prisoners of the division between the world of objects and that of sub- 
jects; instead of making the distinction between subjects and objects, we shall speak 
of associations between humans and nonhumans; the term thus includes both the 
old natural sciences and the old social sciences. (Latour 2004: 237-238). 


Latour’s approach is mostly about expanding the definition of the social to include elements 
and processes that are alterities, which he claims are also constitutive for sociality. In this 
way he hopes to resolve the confusion caused by the dichotomies of traditional social science 
terminology and dismantle their vertical hierarchies. This adds a new perspective to the 
characterization of social orders, for sociality for Latour is not primarily about the vitality 
of social relationships. More importantly, human societies create structures that are not 
founded solely on interaction and coexistence, but also on communication between non- 
human or “asocial” moments that are just as important to social stability as the practices of 
human members of society (Schulz-Schaeffer 2000: 209). To guarantee the endurance of 
social relationships it is therefore not enough to rely only on human collaborations. Rather 
the focus must be shifted, as Latour also says, from the human to the non-human. Key to 
the social theory of actor-network-theory is that all social interactions must be delineated as 
a coevolutionary result of society, nature, and technology (ibid.). 

At the same time, something in the symmetry of human/non-human has not been 
thought to the end, for the hyphen that connects and separates the actor-network remains 
undefined. While indeterminacy makes polyvalence possible by expanding social relation- 
ships to include the non-social, an explication of their relation and its modalities is still 
necessary. This is all the more true when contrasted with interpersonal relationships, which 
are founded on communication and cooperation and for their part formulate the conditions 
needed to enter into connections, whether social or asocial. It is also necessary to differ- 
entiate between relations and relationships, for every relationship references a core alterity 
that exudes an attraction that draws us into a relationship. From the beginning we are deal- 


ing with an arch-passivity (Blanchot) that determines the relationship according to what 


it refers to, while relations are formal assignations such as those that define mathematical 
functions. These cannot be conflated into one, so that in the ecologies of the human and the 
non-human and their concurrent ensembles, we must differentiate between human-human 
relationships, human-object or human-technology relationships, and object-object relations. 
xoLvwvia refers in the main to the first two, while the latter belongs at best to the more or 
less contingent complex of conditions. 

‘This makes clear the internal problems of network theories as formal conceptualiza- 
tions of xowwvica. They imply an exclusion that proves inconsistent in that no network can 
recognize that it is a network, as it has no consciousness of where it begins or ends, whereas 
knowledge and self-knowledge have always been intrinsic elements of social interconnec- 
tions. No holism can see itself as a whole and delineate its limits, so that the supposed 
relational egalitarianism of the net and its equally supposed egalitarian participative nature 
make up their own kind of atrophied stage of the social world, because they can encom- 
pass only relations, which however fall under the same formal schemata and thus tend not 
to be able to concretize their practices.’ xowwvia, like sociality in general, consequently 
shrinks to become a restricted code. Relationality and participation alone are not enough 
to compensate for the maleficium of the net. Firstly, because the non-human can still only 
be conceived from the perspective of the human, and the post-human extension takes place 


» a 


by the will of humans, and, secondly, because terms such as “collective,” “association,” or 
“participation” prove to be symptomatic of a social condition that is in the process of restruc- 
turing to become solely technological. What become conspicuous, and what systematically 
falls through the cracks of these categories, are the types of relations that seem indispensable 
for the constitution of the social. To these belong, alongside trust, the ethics of the gift and, 
in particular, the political ethics of justice, which cannot be fulfilled by taking apart and 
taking part because xowwvia, like democracy, is first and foremost an ethical and a legal 
concept. 

The truly explosive nature of the question is found at the site where Latour’s terms 
are radicalized within cultural and media studies, simultaneously eradicating their genuine 
social-philosophical impetus (Thielman Schröter 2014: 148-158). Even if Bernhard Siegert 
and Lorenz Engell (2013: 5) have called actor-network-theory “one of the most interesting 
conjunctions of the cultural sciences and media studies in recent years,” it is equivalent to an 
abuse of categories when “symmetrical anthropology” is cast as an “actor-media-theory” and 
the idea of networks is glibly applied to “medial actors” and their connectivity (Schüttpelz 
2013). These are understood primarily as “cultural technologies” that first broadcast art, 
science, history, or thought and observation—a totalization that ignores that every tech- 
nological invention is already in the arena of cooperation, which in turn is unthinkable 
without xowavia. When today media studies and cultural studies frame themselves as the 
theoretical avant-garde and assume a materialistic “apriority of technicity” and therefore 
also an implicit technological anthropology with all of the attributes of determinism, then 
it should be remembered that they do not possess an adequate concept of the social. Media 
and cultural studies seem as it were to be blind in the social eye. This broaches the question 
of the fate ofxorvwvio under technological and medial conditions, as well as its representa- 
tion in networks whose graphs can perhaps connect communication tools, technical objects, 
archives, smart things, operative systems or gadgets and, especially, algorithms, artificial 
figures and techno-imaginative androids and even sometimes also people, but do not come 
close to that which makes connections special and makes them connect us. 

This is betrayed by the most common terminology, particularly “operativity” 
“circuits,” “interfaces,” and the like. These terms correlate strangely with the aforemen- 
tioned reveries of counter-cultures at the time of the watershed invention of the PC, a 
caesura in computer history. In the meanwhile, these computer environments have received 
exponential upgrades including ubiquitous sensor technology, databases, digital statistics, 
and artificial intelligence, while remaining oddly alienated socially. Taken on by theo- 


ries of media and culture, this terminology has also been taken for a ride, or corrupted by 


7 “Literally there is nothing but 
networks, there is nothing in 
between them, or, to use a meta- 
phor from the history of physics, 
there is no aether in which the 
networks should be immersed. 
In this sense AT is a reductionist 
and relativist theory, but as | 
shall demonstrate this is the 
first necessary step towards an 
irreductionist and relationist 
ontology.” (Latour 1996: 372) 
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these disciplines. The so-called social media in particular—which in reality are singularly 
asocial— are the perfect illustration of what happens when all there is networks with noth- 
ing “between them” —no “aether” and also no other binding media. All that connects them 
is formal graphs as well as “nodes” and “edges” which in turn cannot be described on the 
foundation of the same graphic network repertoire but do reveal the social ruin created by 
pure formalism. One must think of xowwvia under technological conditions as a network, 
which conversely makes clear that it has lost its essential essence, namely acting as a “force” 
that allows for “assembly” and embodies the “common,” and has instead abdicated it to an 
“image” or an external structure. 

In fact, although technological network metaphors seem to underscore connectivity, 
they preserve that which is insufficient in them, because they form an abstract concept that 
is modelled by diagrams that only portray their temporary and local connections without 
providing any information about what kind of linkages they are, or via which paths these 
linkages come to be. It is decisive that the operationalist framework of technologies, as well 
as the relational model of network theories, do not even come close to being able to con- 
ceptualize a non-formalist, which always means performative concept of relationship, which 
is more than just an interface and an interaction-network. xowwvia in this conception 
is nothing more than a blueprint that can be laid over all kinds of social processes like a 
map, but that moves further and further away from us the farther its formalism progresses. 
For as a blueprint, it offers no more than a virtuality; a plan that says what could be built 
but not how it should be built on a practical level. At best it gives those who know how to use 
it a vague orientation, just as it confuses those who cannot decipher it. Instead, we should 
hold fast to the task of granting a specific reality to the social and with it to xowwvia as a 


practice. 


POSTSCRIPTUM 


The seemingly abstract reflections on the constitution of sociality and its central con- 
cepts, such as communication, trust, cohesion, reliability, and the like—in one word, the 
xowwvia.— directly concern the issue of living together under technological conditions, 
with possible encounters between humans and nonhuman beings including decision- 
making robots, avatars, or other artificial creations, as well as with autonomous agents such 
as self-driving cars. Posthumanism has placed such encounters under the radical perspec- 
tive of their ontological egalitarianism. The fact that we (as human beings) are together 
with nonhuman entities in the same way that we are together with other human beings 
decenters our primary role in the world and disempowers the claims of classical humanism 
(Mersch 2022). In turn, the critical attitude of decentering implies a rejection of domination: 
it devalues our power relations and, according to Latour (1996, 2005), places our actions 
back into a network of equality among different actors, human and non-human. But the 
critique does not touch the problem of “relationship” (which is more than just a “relation”) 
and thus of sociality and the question of whether the human—non-human relations can 
be considered under the same auspices and values as human—human relationships. The 
terms “relation” and “relationship” — or in German: Relation and Beziehung, which contains 
the word ziehen or “to pull” —need to be distinguished, as the first denotes only a formal 
function, while the second implies a force that asks or invites me to relate. Therefore, the 
suspicion arises that we behave differently towards other humans than towards objects or 
non-human actors— no matter how much they are endowed with “intelligence” or an auton- 
omous decision-making capacity. Obviously the social, in terms of “being with,” means 
something completely different pertaining to human beings than in relation to nonhuman 
actors. Moreover, social facts are first and foremost not a matter of power, but primarily of 


“encounter.” Thus, we can also raise the same question from another angle: What does it 


mean to meet, to interact with, or to develop feelings and desire towards technical beings 
that are neither creatures (animals or plants) nor humans, i.e., that are not life-forms that 
share the experience of otherness and mortality? 

Certainly, psychoanalysis has dealt with the notion of projection, which seems 
to correspond to our psychic nature as it constantly approaches pathology, fetishism and 
inappropriate anthropomorphism. In this respect we are surely able to engage with objects 
emotionally, lending them attention and appreciation; we may even love them like the 
automatic voice in the film Her (2013), but this signifies neither a reciprocal relationship 
nor a social connection, for sociality depends on mutual recognition, on the mirror expe- 
rience of equality, on cooperation and the need for community with its cultural norms, 
rules, and values, and above all on the asymmetrical relation of responsivity that allows for 
self-understanding and gives birth to true 
responsibility. Can this apply in the same 
way to technical artifacts like avatars as it 
does to our fellow humans, to our “neigh- 
bors” (Nachste)? Is confusion between artifi- 
cial beings and real persons conceivable? Is a 
non-pathological longing for acceptance by 
an artificial being, triggering in us the desire 
for response and attention, even possible? 
What if we encounter avatars that osten- 
tatiously turn away from us, ignore us? Do 
we feel insulted or simply indifferent? And 
what does non-indifference mean pertaining 
to things; what kind of sociality is it if we 
prefer to surround and converse solely with 
avatars or other artifacts? Can, under such 
conditions, the tremendous loneliness of 


self-encounter ever be overcome? 


ting time 


The questions at stake tackle the = 
meaning of the social in the face of a disso- 
lution of any distinction between the human 
and the non-human, which already implies 
the complete dissolution of any meaning of 
humanity as well. These ques- 

tions arise concretely against the 


background of so-called “social” FIGS. 1, 2 


media, the internet as a primor- FUTURE PROJECTIONS OF VIRTUAL MEETINGS. 


dial tool of exchange, engage- 
ment, and interaction, and, 
above all, a pervasive “avatariza- 
tion” of personal relationships. Can the concept and experience of the social then retain the 
same significance as that which we once gave to various expressions such as “community,” 
“society,” or even “congregation” or “commune”? These have always had a political as well 
as a religious and legal meaning; but since there are technical artifacts that seem to speak, 
act, and think— obviously all these concepts are misleading —the psychoanalytic term of 
projection, which has always described a subconscious relation, is no longer sufficient to 
do justice to the situation. Rather, what is needed is a deeper analysis of different modes of 
encounter (instead of modes of “existence”), which includes the broader questions of what it 
means to be “in communion” with artificial beings and whether it is possible to enter into 
political relations with them or to cooperate with them emotionally. The obvious problem, 
then, is: What remains of the social when our primary “society” consists almost entirely of 


technical relations using technological beings? 
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FIG. 3 
MARC ZUCKERBERG MIRRORING HIMSELF 
IN HIS VISION OF THE “METAVERSE.” 


8 See for instance 
https://www.youtube.com/ 
watch?v=b9vWShsmE20 
(accessed February 10, 2022). 

9 Ibid. 2:10-2:35. Curiously enough, 
in the video, Zuckerberg has 
already completed his transfor- 
mation process into an avatar: he 
is the avatar that he is promoting. 

10 See https://abbavoyage.com/ 
(accessed February 10, 2022). 
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One could dismiss this elaboration regarding possible relations or relationships with ava- 
tars—or “rendez-vous,” as Jean-Luc Nancy (2021) calls them—as exaggerated or unreal 
and relegate it to the narratives of science fiction. However, since Mark Zuckerberg wants 
to expand Facebook from a “social medium” to what he calls a “Metaverse” (following the 
model of the online game Second Life, which will have been its naive design precursor, as 
it were) that consists of a complete virtualiza- 
tion of social life, so that we transform our- 
selves into photorealistic avatars who meet 
with other avatars in imaginary landscapes 
or workspaces, the question of face-to-face 
encounters in particular has gained a new 
topicality and dimension.’ It will arise with 
real urgency in the near future. Will such 
virtual encounters harm or affect or even 
destroy our social abilities, which depend on 
the indispensability of other humans being 
present in terms of direct contact, physical- 
ity, and the desire to be 
touched—rather than 
simply be connected? 
While Zuckerberg in 
his video promotion of 
the Metaverse explicitly 
emphasizes the “feel- 
ing of presence” as the 
“defining quality of the Metaverse”— “you are going to really feel like you're there with 
other people, you'll see their facial expressions, you'll see their body language (...), all the 


”9 — these facial 


subtle ways that we communicate that today’s technology can’t quite deliver 
expressions and body language are simple perceived from a distance and reduced to visual 
communication, simultaneously subtracting the essence of a true encounter. Every detail, 
the entire private ecosystem, is subject to our defining power and control, so that we are 
dealing with a fundamental attack on reality, which is replaced with a second, artificially 
constructed one. This corresponds to the fact that to make the illusion possible we also 
entrench ourselves behind VR headsets, which thus become an emblem of a displacement of 
the world, sealing us off from the gaze of the Other. Coincidence, deviance, the unpredict- 
able and unavailable as necessary dimensions of the social are thus carefully eradicated and 
erased. The future of social Amedia clearly lies in the establishment of an ongoing “digital 
playground.” 

Unlike robots or autonomous agents, avatars are screen beings; they form figures 
that are virtual representations, reduced to mere visuality. They rely on digital simulations 
with computed expressions and gestures, whereas interpersonal encounter is based on bodily 
co-presence with attributes such as fragility, vulnerability, but also defensiveness, shame, 
or shyness, attributes that remain unknown to “avatarian” representations. Their technical 
realizations are limited to scans or motion-capture systems, which are supposed to guar- 
antee an optimized performance of their “as-if,” the prototype for which originated from 
animated films or computer games. However, the contorting of certain facial muscles or 
the execution of specific hand movements alone do not reveal any “motion” of emotions, no 
inner movement, let alone moods and their ambiguities. 

Yet it is not only avatars with whom communication is planned, but also holograms. 
Beyond screen representations, experiments are being carried out on stage as in the planned 
“ABBA comeback tour” in London starting in May 2022.1? Whatever the “ABBAtars” used 
in these events are, they present themselves as so-called “real avatars,” separated from the 


audience by the “fourth wall” of the show. Thus, they remain, as in the envisaged Metaverse 


meeting, virtual images that have no body and thus no body feeling, no body scheme, and 
therefore also no expressive qualities. Accordingly, there is no closeness or bonding, because 
if we want to reach for them, we reach through them into the void. The encounter thus exists 
only superficially, an imitation or pure shadow of real physical touch with all its imponder- 
ables and subtleties. Instead, it satisfies the criterion of similarity, of deception—the form 
of imitation and pretense that has dominated digitization right from the start. Look, for 
instance, at Alan Turing (1950), who, in order to answer the question of whether or not 
machines could be intelligent, proposed an imitation game that aims at nothing less than to 
lure us into a trap (Longo 2019). 

The relationship with Others who, unlike mere images or surfaces, remain, as 
Emmanuel Lévinas has described, pure “enigmas” that address us, haunt us, or challenge 
us, is something completely different from any as-if or virtual game, containing as it does 
possibilities that may or may not occur. Such a relationship implies a “bindingness” that 
cannot be evaded, which consequently always already contains commitment and obligation, 
whereas virtual representations induce nothing like that. Can sympathy arise under the 
condition of virtuality? Trust? Pity? Care or concern? Or desire and love? The designers and 
engineers— the self-proclaimed elite of the digital age—usually answer these questions 
with the suggestive counter-question: “Why not ...?” In this way, they already manifest that 
they have lost touch with others, with what is meant by a “social relationship” in its emphatic 
sense, because they judge solely from the perspec- 
tive of a methodical solipsism, as if we were fun- 
damentally detached and enclosed in our world, 
and as if it were our decision to whom we want to 
turn, instead of having always already interacted 
with “others,” our fellow human beings. Igno- 
rance or heedlessness is not an option in the social, 
because we are only human as far as we belong to 
others. On the other hand, to be in the company 
of avatars, to deal with them “like others,” corre- 
sponds to a primary indifference—the awareness, 
that none of it is “real” —that has already 
gamified the social and subjected it to 
mutual staging and self-presentation. This FIG. 4 


applies equally to the schizophrenogenic SCENE FROM THE TRAILER FOR THE ABBA 
confrontation with ourselves as avatars, TOUR, PLANNED FOR MAY 2022 IN LONDON. 


who—unlike the doppelganger—are 
hardly able to unsettle us, because as bod- 
ies we always know when we are among or 
with other bodies, or only with ghostly apparitions. But the fact that the Metaverse primar- 
ily revolves around our self-presentation and egocentrism, sublimated by auto-avatarization, 
already underscores the fact that others are at best vehicles to equip our stage and enable our 
performances. The social shrinks structurally to the functions of our own ego-performance 
(as Zuckerberg shows). 

Posthumanists and some media and cultural studies scholars who instead assume a 
future destabilization of our well-accustomed certainties —the “disappearance of the real” 
or the dissolution of classical distinctions such as that between being and appearance, orig- 
inal and copy, truth and lie, or subject and object, nature and culture, and human and 
non-human—have not seriously learned their own deconstructive lessons and have not 
understood their own critiques. The collapse of differences that they proclaimed applies at 
best to screen media or the technical illusionary machinery. Today’s theater, after a phase of 
extreme asceticism in the 1970s and 80s, has also turned to technological fascination and 
is increasingly striving to transform itself into spectacle. But in the face of the screens or 


technical effects, the pleasure in deception dominates, because it is solely based in the eye 
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n_e 
11 See https://this-person-does- 
not-exist.com/de 
(accessed February 10, 2022). 


and the ear of the beholder, so that the prognosis of the collapse of differences corresponds 
more to a wishful image than to reality. Admittedly, we are deceivable and also deception- 
ready beings, with, as Friedrich Nietzsche expressed in his early text “On truth and lie in 
the extra-moral sense” (1873), a pronounced will to lie, to fiction—but this can only succeed 
where we rely on technical reproducibility and its assigned senses, while in turn all senses 
of “distancelessness” such as haptics, tactility, smell, and taste are simultaneously excluded, 
for it is these that structure our experience of reality as an experience of existence in the 
first place (Mersch 2002: 30-42). We do not, of course, fail to recognize the existence of 
“absolute deception” through DeepFake: consider, for example, the website “This person 
does not exist,” with nothing but imaginary photographs of non-existent persons generated 
by Artificial Intelligence, which remain indistinguishable from “real” ones." The danger 
they pose is great, but they primarily expose us to a visual Turing test, which, however, 
already presupposes the senses of sight and hearing, cutting off any synesthesia. Thus, in fact, 
any technical illusionism is governed by a reductionism— not only aesthetic—which, as 
magic, always tries to erect a dazzle in front of us in order to render us blind to transparency 
and critical reflection. 

In opposition to this, the aforementioned “existential senses,” above all touch, offer 
a corrective. They likewise teach a correction vis-a-vis the avatarization of the social, so 
that the alleged collapse of differences fails to take place. Instead, its exaggeration can only 
take hold where we allow the technically superimposed eyes and ears to usurp our ways of 
experiencing the world and ourselves, that is, where the power fantasies of technology have 
taken hold in such a way that we distrust the sovereignty of perception and find ourselves 
outside of what must be understood as an actual “bodily encounter.” This encounter proves 
to be just as constitutive for the facticity of the social as for alterity and responsivity as the 
elementary references to others in terms of the always elusive otherness. Avatars instead deny 
any alterity, or rather: they are—unseeing and never truly seen—“apparent others” (in the 
double sense of appearance and semblance, of visual gloss and deception), incomparable 
with human companions, reduced to the mathematical modeling of their skin as well as the 
computation of sounds and movements, whose intonation, expression, and gesture— how- 
ever perfectly imitated — remain clumsy and who, in one word, are not “aliens” to us, but at 
best “alienating”. 

In contrast, in one of his last conversations with the art-historian Carolin Meister, 
Jean-Luc Nancy (Nancy/Meister 2021) insisted that “encountering each other” presupposes 
a “togetherness” that is always already embedded in social reciprocity. This turns out to be 
rudimentary in the confrontation with avatars. According to Nancy, encounters possess 
something “irreducible” and “inexplicable” (ibid.: 9); they are event-like, because we can 
neither reckon with them nor anticipate them. “Encounter does not constitute precisely a 
relation, neither connection nor communication, but rather a challenge to all these terms” 
(ibid.: 11)—here understanding the notion of relation as a formal relationality. Perhaps, 
therefore, it would be more accurate to say that the encounter provokes what technically 
appears to be an interaction; for its event, Nancy continues, is “entirely on the side of the 
unexpected, the accidental”; consequently, it is something “that happens to us” (ibid.: 16) 
and thus takes place at the level of the “It” — or rather, “between fragments, moments, or 
aspects of the ‘It” (ibid.: 24). Every encounter, therefore, proves mysterious, incomprehen- 
sible, and impenetrable: “[It] will never attain clarity about itself” (ibid.), for it is— through 
the Other and its unknowability, its invisibility — always resistant to us and to our desires, 
remaining beyond what can be known or apprehended by consciousness: “The two of an 
encounter have before them and within them—and likewise between them—the entire 
human population” and indeed “humanity” in general (ibid.: 31). 

This cannot be said of an interaction with avatars, even for “real-time” avatars that are 
based on an instant avatarization of ourselves. No matter how much it may seem as if some- 
thing unexpected lies in their appearance as well, what appears turns out to be thoroughly 


controlled by the camera we use and pre-determined by the computational power of the 


underlying program. ‘They lack the subtlety of physical tension, nuance, and traces of an 


unconscious, because they can only represent— within the realm of visual resolution— what 


can be digitally calculated. Founded on operational functions, they also lack the temporal- 


ity and history that would first and foremost give them depth. It is this lack of depth that 


thwarts any “encounter” in the proper sense, which would unintentionally reverberate on 


the experience of the persons the avatars purport to represent. Therefore, as Nancy states in 


his conversation with Meister, “(p)erhaps it must be said that there is no encounter within 


mathematics (...), and that there is equally no encounter between the mathematical or math- 


ematized sphere and the life-world or the world of language” (ibid.: 45). The foregrounding 


of non-human behavior thereby blots out the background of human intentions and their 


vagueness and unfathomability in such a way that, disguised as avatars, we are never present 


and thus never able to meet entirely in the sense of involving all the senses, precisely also the 


hidden ones and their libidinal or confused signatures. It is this that constitutes the charm 


or magic of an encounter in the first place. 


This also means that the replacement of the social with avatarized contact, as envi- 


sioned by future internet platforms such as the Metaverse, leaves little of what constitutes 


sociality— much less than already remains today of understanding under the umbrella of 


ubiquitous communication media. What was once intended to enable connection proves, in 


the context oframpant hate messages, to be the exact opposite: an excess of social polarization, 


and this is even more true for virtual realms such as the Metaverse and similar Silicon Valley 


creations. Thus, when Nancy and 
Meister say “I am all the more me 
when I encounter you” (ibid.: 76), in 
order to indicate that social bridges 
exist only where we are already 
together “in relationship,” where we 
are committed to others from the 
beginning, the statement must be 
read in the context of an avatariza- 
tion as: “I am all the more me when 
I play ‘me’ in front of ‘you’ as my 
audience.” As avatars we “perform 
each other,” rather than encounter 
the other. The two propositions for- 
mulate separate worlds, which ren- 
ders the rift, the gap or difference, 
between two understandings of the 
collective absolute. In this situation, 
the significance of xowavia can no 


longer be reconstructed. 
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